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6th Annual CMMI Technology Conference & User Group 


Denver, Colorado 


November 11 - 17, 2006 


TUTORIAL SESSIONS - MONDAY, NOVEMBER 13, 2006 


TRACK 1 


e 1A1 Tutorial, CMMI v1.2 Tutorial, Mr. Mike Phillips, SEI 
e 1C1 Tutorial, Are We Ready for CMMI? If Not, Let’s Fix Ourselves, Mr. Al Florence, The MITRE Corporation 


TRACK 2 
e 1C2 Tutorial, Calculating CMMI-based ROI: How, What, When and Why?, Mr. Rolf Reitzig, Cognence, Inc. 
TRACK 3 
e 1A3 Tutorial Addressing High-Value Project Management Activities Using Process Simulation & Supporting CMMI for Process 
Areas, Dr. David Raffo, Portland State University 
e 1C3 Tutorial, Into the Future, Part 1: Process Definition on Steroids, Ms. Kathy Reid, 96th Communications Group/TYBRIN, USAF 
TRACK 4 
e 1A4 Tutorial, Integrated Project Management (IPM) - The CMMI and Collaborative Product Development, Mr. William Deibler, II, Software Systems Quality 
. aan CMMI Crash Course: What the SEI Won’t Teach You, Mr. Hillel Glazer, Entinex, Inc. 
TRACK 5 


e 1A5 Tutorial, A Practical Guide to Implementing Levels 4 and 5, Dr. Rick Hefner, Northrop Grumman Corporation 
e 1C5 Tutorial, Basics of PSP and TSP for Systems Engineering, Mr. James McHale, SEI 


TRACK 6 


e 1C6 Tutorial, Requirements Engineering: A Practical Approach to Modeling and Managing Requirements, Mr. William Deibler, II, Software Systems Quality 
Consulting 


TRACK 7 
e 1A7 Tutorial, Lessons Learned in Implementing CMMI in a Large Systems Engineering Organization, Mr. Michael Kutch, Jr., SPAWAR, Charleston 
TRACK 8 


e 1A8 Tutorial Lean Process Improvement, Mr. Jeffrey Dutton, Jacobs Technology 


TUESDAY, NOVEMBER 14, 2006 
KEYNOTE 
e VADM Walter B. Massenburg, USN, Commander, Naval Air Systems Command CMMI® Into the Future 
e Mr. Clyde Chittister, Chief Operating Office, SEI 
e Mr. Bob Rassa, Director, Systems Support, Raytheon Company 
EXECUTIVE PANEL 


Moderator: Mr. Mark Schaeffer, Director, Systems & Software Engineering, OSD (AT&L) 
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e Mr. Roger Fujii, Vice President, Communications and Systems Technology, Defense Mission 
e Mr. Dev Banerjee, Division Director, Systems & Flight Engineering, Integrated Defense Systems St. Louis, The Boeing Company 
e Mr. Mark Schaeffer, Director, Systems & Software Engineering, OSD (AT&L) 


TECHNICAL SESSIONS 
TRACK 1 - CMMI and Process Improvement 


e The Continuous Representation: Cultivating Process Improvement from Grassroots Efforts Ms. Jennifer Turgeon, Sandia 
National Laboratories 

e Who Needs that Process Stuff Anyway? A Practical Example of Process Discipline to Improve Deployment, Mr. William Borokowski, Jr., Raytheon Company, 
Missile Systems 

e 2D1 FASTrack to CMMI, Mr. Al Florence, The MITRE Corporation 

e 2D1 Implementing CMMI-based Process Improvement Using RUP, Mr. John Miller, Cognence, Inc. 


TRACK 2 - High Maturity 


e 2C2 Business Value of CMMI Level 5, Dr. Gene Miluk, SEI 

e 2C2 Demonstrating CMMI-based Process Improvement, Ms. Elizabeth Clark, Lockheed Martin Corporation, IS&S 
e 2D2 Report to OSD: NDIA Top 5 Software Issues, Mr. Bob Rassa, Raytheon Company 

e 2D2 Right Turns and Wrong Turns to Level 5, Mr. Timothy Davis, Raytheon Company, Missile Systems 


TRACK 3 - Appraisals 


e 2C3 Improving SCAMPISM Efficiency, Mr. Roland Weiss, Lockheed Martin Corporation 

e 2C3 SCAMPI A Appraisals - Preparation in100 Hours; Execution in a Week, Mr. Ronald Ulrich, Northrop Grumman Corporation, Mission Systems 

e 2D3 ISO and CMMI: How to Perform SCAMPI Appraisals and ISO Audits in a Syncronized, Simultaneous, Stupendous One Fell Swoop, Ms. Margaret 
Glover, SEI 

e 2D3 Experience Report - An ISO/IEC 15504-conformant SCAMPI Appraisal, Mr. David Kitson, SEI 


TRACK 4 - ROI & Benefits of CMMI 


e« 2C4 CMMI-based Process Improvement: How and When Does Success Happen?, Dr. Dennis Goldenson, SEI 

e 2D4 The Rosetta Stone CMMI ROI Model - Translating Between ROI, CMMI and Metrics, Mr. Fionbarr McLoughlin, 
University of Limerick 

e 2D4 Quick Win Strategies in CMMI Transition, Ms. Angel (Qi) Liu, Motorola GSG, China 


TRACK 5 - CMMI Extensions: Acquisition 


e 2C5 CMMI for Acquisition: Introducing the CMMI for Acquisition Constellation, Ms. Kristen Baldwin, Office of the 
Secretary of Defense 

e 2D5 The Hidden Truths of Being a Successful Acquirer, Dr. Hubert Hofmann, General Motors 

e 2D5 When CMMI-AM No Longer Suffices, Mr. Claude Marais, TPI, Inc. 


TRACK 6 - Systems Engineering 


e 2C6 Developing Systems Engineering Processes from Existing Software Processes, Ms. Lee Sheiner, Georgia Tech 

Research Institute 
e 2C6 Extending the Team Software Process (TSP) for Systems Engineering, Mr. Timothy Chick, Naval Air Warfare Center, Weapons Division 
e 2D6 No Doubt: The LRAS3/FS3 Story of Mission Assurance, Mrs. Jill Brooks, Raytheon Company 


TRACK 7 - Lean, Six Sigma and Agile 


e 2C7 Rapidly Defining a Lean CMMI Maturity Level 3 Process, Ms. Julie Kellum, Hewlett Packard Company 

e 2D7 Using Business Process Management Technology to Implement CMMI-compliant Agile Software Development, Mr. Robert Moore, III, Business 
Transformation Institute 

e 2D7 Using “Voice of Customer” Tools to Advance Organizational Innovation and Deployment, Mr. Donald Corpron, Northrop 
Grumman Corporation 


TECHNICAL SESSIONS - WEDNESDAY, NOVEMBER 15, 2006 


TRACK 1 - CMMI and Process Improvement 


3A1 Communication Loops Within the CMMI, Mr. Shawn Presson, Apogen Services, LLC 

3A1 Knowledge Management at the Heart of CMMI ML3 at RAFAEL, Dr. Amir Tomer, RAFAEL, Ltd. 

3B1 Organizational Synthesis: CMMI The Glue that Binds, Ms. Eileen Arnold, BAE Systems, L&A 

3B1 Operations and Maintenance - More Than Just a Scaling Issue, Mr. Laurence Gill, BAE Systems 

3C1 Program Management Office (PMO) and CMMI - Paving the Road to Optimized IT Value: Case Study, Mr. Terry Jabali, Applied Enterprise Dynamics, 
Inc. 

e 3C1 How to Sell CMMI to Your CEO in His Own Language, Ms. Lemis Altan, Cognence, Inc. 
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e 3D1 Make Middle Managers the Process Owners, Mr. Tim Kasse, Kasse Initiatives, LLC 
e 3D1 You Shouldn’t Have to Be an Engineer to Understand Your Process, Ms. Leticia Santillan, Raytheon Company, Missile Systems 


TRACK 2 - High Maturity 


e 3A2 What is the Performance Effect of the CMMI-based Process Improvement in Motorola?, Ms. Angel (Qi) Liu, Motorola GSG, 

China 

3A2 Learning to Count - Do You Understand Why Level 3 Comes Before Level 4 Which Comes Before Level 5?, Ms. Nancy Raymond, Raytheon Corporation 
3B2 Making OID Effective, Ms. Diane Mizukami Williams, Northrop Grumman Corporation, Mission Systems 

3B2 When OID Crosses Time Zones, Disciplines and Customer Base, Ms. Laurie Haack, Raytheon Company 

3C2 Looking Beyond Quantitative Defect Management, Mr. Anil Midha, BAE Systems 

3D2 Statistical Control of System and Software Design Activities, Dr. Richard Welch, Northrup Grumman Corporation 

3D2 I Have Had My CMMI Appraisal... What Do I Do Now? How to Establish a Process Improvement Work Breakdown Structure, Dr. Aldo Dagnino, SAS 
Institute 


TRACK 3 - Appraisals 


3B3 Lessons Learned from Conducting Scampi B and C Appraisals, Mr. Itzhak Lavi, Israel Aircraft Industries, Ltd 

3B3 Practical Advice on Picking the Right Projects for an Appraisal,Mr. Robert Moore, III, Business Transformation Institute 
3C3 Lessons Learned: Getting Ready for a SEI Maturity Level 5, Mr. Dennis Ring, Lockheed Martin Corporation, IS&S 

3C3 Indicators of Reliable SCAMPI Appraisal Results, Mr. Bruce Boyd, The Boeing Company 

3D3 Project Implementation Strategies in the CMMI, Dr. Rick Hefner, Northrop Grumman Corporation 

3D3 Patterns: An Approach for CMMI Adoption, Dr. Rick Hefner, Northrop Grumman Corporation 


TRACK 4 - ROI & Benefi ts of CMMI 


e 3A4 Evaluating the Impact the QUARS Requirements Analysis Tool Using Simulation, Dr. David Raffo, Portland State 
University 
e 3A4 Measuring the Value of Systems Engineering Processes, Mr. Allan Mink, II, Systems Value 


TRACK 4 - Measurement 


e 3Bé4 Practical Earned Value Management Integrated with CMMI-compliant Processes, Dr. Donn Milton, Pragma Systems 
Corporation 

3B4 Extracting Strategic Value from CMMI, Dr. Rick Hefner, Northrop Grumman Corporation 

3C4 Performance Benchmarking Consortium, Mr. Mark Kasunic, SEI 

3D4 2006 State of Software Measurement Practice Survey, Mr. Mark Kasunic, SEI 

3D4 Measures and Lessons from a CMMI Guided Process Improvement Project, Dr. Ruth Buys, 3H Technology, LLC 


TRACK 5 - CMMI Extensions: Acquisition 


e 3A5 Using CMMI-AM to Measure AF System Program Office Implementation of Systems Engineering, Mr. Ian Talbot, US Air Force, Air Armament Center 
e 3A5 Understanding and Leveraging a Supplier’s CMMI Efforts:A Guidebook for Acquirers, Mr. Lawrence Osiecki, OUSD 
(AT&L) SSE/SSA 
e 3B5 SCAMPI B’s for Contract Monitoring - A Case Study of the Mission Planning Enterprise Contractors, Ms. Lorraine Adams, SEI 
e 3B5 DCMA CMMTI-related Data Collection: A Maturity Model-based Reflecction of Practice Strengths and Weaknesses 
Mr. John Eget, DCMA 
e 3C5 Practical Use of DCMA’s CMMIbased Risk Management Methodology, Mr. Gary Sigmund, DCMA 
e 3C5 The Issue of Performance - Why Do You Need a Maturity Level 5 Supplier?, Mr. Tim Kasse, Kasse Initiatives, LLC 


TRACK 5 - CMMI Extensions: Services 


e 3D5 CMMI for Services: Introducing the CMMI for Services Constellation, Mr. Craig Hollenbach, Northrop Grumman Corporation 
e 3D5 CMMI Implementation in Manufacturing Processes, Mr. Itzhak Lavi, Israel Aircraft Industries, Ltd. 


TRACK 6 - Systems Engineering 


e 3A6 Risk Management Beyond CMMI Level 5, Dr. Rick Hefner, Northrop Grumman Coproration 

3A6 CMMI Opens the Gate - The Practical Relationship between CMMI and Stage-Gate Decision Models, Mr. Andrew Cordes, ABB US Corporate Research 
e 3B6 Mission Assurance of Software Suppliers - A Partnership Between Software Supplier Managers and Quality Engineering, 

Mrs. Jill Brooks, Raytheon Company3B6 Systems Engineering Revitalization Using CCMI, Mr. Michael Kutch, Jr., SPAWAR, Charleston 


TRACK 6 - CMMI v1.2 and Beyond 


e 3C6 CMMI V1.2 Overview, Mr. Mike Phillips, SEI 

e 3C6 SCAMPI 1.2 Updates, Dr. Jack Ferguson, SEI 

e 3D6 Strengthening the Credibility of Level 4/5 Appraisals, Mr. Will Hayes, SEI 
e 3D6 Lead Appraiser Training for V1.2, Dr. Jack Ferguson, SEI 


TRACK 7 - Lean, Six Sigma and Agile 


e 2C7 Rapidly Defi ning a Lean CMMI Maturity Level 3 Process, Ms. Julie Kellum, Hewlett Packard Company 

e 2D7 Using Business Process Management Technology to Implement CMMI-compliant Agile Software Development, Mr. Robert Moore, III, Business 
Transformation Institute 

e 2D7 Using “Voice of Customer” Tools to Advance Organizational Innovation and Deployment, Mr. Donald Corpron, Northrop 
Grumman Corporation 
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e 3C7 Lean Process Design - A Concept of Process Quality, Mr. David Card, Q-Labs 
TRACK 7 - CMMI in Small Settings 


e 3D7 Easing Project Planning for Small Programs, Mr. Kenneth Weinberg, Raytheon Company 
e 3D7 The Road to Lean CMMI in a Small Federal Organization, Ms. Susan Bassham, US Army, Aviation and Missile Command 


Technical Sessions - Thursday, November 16, 2006 


TRACK 1 - CMMI and Process Improvement 


e 4A1 Moving from Level 2 to Level 3 Risk Management, Ms. Susan Byrnes, Natural SPI, Inc. 
e 4A1 Interfaced vs. Integrated Process, Mr. Robert Rice, BAE Systems 


TRACK 2 - Sustaining Improvement 


e 4A2 Motivating the Team to Continuously Improve After the CMMI Rating, Ms. Leticia Santillan, Raytheon Company, Missile Systems 
e 4A2 King of the Mountain - Staying Atop Mt. CMMI After the Initial Assent, Mr. Timothy Davis, Raytheon Company, Missile Systems 
e 4B2 Sustaining CMMI Compliance, Dr. Rick Hefner, Northrop Grumman Corporation 

e 4B2 Maintaining Institutionalization After the Appraisal Team Leaves, Mr. John Johnston, PMP, BAE Systems 


TRACK 3 - Implementation Guidance 


e 4A3 An Enterprise Approach to Process Improvement - Ms. Linda Kovar, Raytheon Company 

e 4A3 CMMI and ISO 9001- A Registrar’s Perspective, Mr. Matt Leiphart, Platinum Registration 

e 4B3 It’s a #$*!>% Mess: How to Leverage Today’s Process into Tomorrow’s Initiatives, Ms.Terri Bryant, Raytheon 
Company, Missile Systems 

e 4B3 Systematically Tailoring the Organization’s Standard Process for a Project, Dr. Kenneth Nidiffer, Systems and 
Software Consortium 


TRACK 4 - Measurement 


e 4A4 Integrating Planning, Control, Measurement & Analysis CMMI Process Areas to Achieve More Successful Software Projects, Mr. Daniel Galorath, 
Galorath Incorporated 

e 4A4 The Unseen Benefi ts of the Change Request Process, Mr. Scott Banks, Georgia Tech Research Institute 

e 4B4 Measuring the Impact of RD and REQM CMMI Process Areas, Dr. Aldo Dagnino, SAS Institute 

e 4B4 Untethered Activities - The Real Reason for Schedule Slips, Mr. Rolf Reitzig, Cognence, Inc. 


TRACK 5 - CMMI Extensions: Services 


e 4A5 Lessons in Integrating Nonengineering Organizations with Engineering Using the CMMI, Mr. Joseph Vandeville, Northrop 
Grumman Corporation 

e 4A5 Engineering in a Non-development Environment, Ms. Kathy King, BAE Systems 

e 4B5 Achieving Delivery Excellence Using CMMI, Mr. Keith Heston, Accenture 


TRACK 6 - CMMI v1.2 and Beyond 


e 4A6 Implications of Combining SAM and ISM in CMMI V1.2, Mr. Mike Phillips, SEI 
e 4A6 A Practical Roadmap for Transitioning to CMMI V1.2, Dr. Rick Hefner, Northrop Grumman Corporation 
e 4B6 Resolving CMMI Issues, Dr. Karen Richter, Institute for Defense Analyses 


TRACK 7 - CMMI in Small Settings 


e 4A7 CMMI Applications in Small Setting Finance Organization for Process Performance Optimization, Dr. Mary Anne Herndon, 
Transdyne Corporation 

e 4A7 Small Project Challenges of CMMI in Large Organizations, Mr. John Bethmann, Concurrent Technologies Corporation 

e 4B7 Why Choose Between CMMI, ISTO, ITIL and Six Sigma? Why Not Leverage All!, Ms. Lemis Altan, Cognence, Inc. 
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SUNDAY, NOVEMBER 12, 2006 
3:00 PM - 6:00 PM Conference Registration Open Grand Mesa Foyer 


MONDAY, NOVEMBER 13, 2006 
5:30 PM - 7:00 PM Reception Atrium Display Area 


TUESDAY, NOVEMBER 14, 2006 


7:00 AM - 5:00 PM Conference Registration Open Grand Mesa Foyer 
7:00 AM - 8:00 AM Continental Breakfast Atrium Display Area 
8:00 AM - 8:15 AM Welcome & Opening Remarks Grand Mesa DEF 


¢ Mr. Sam Campagna, Director, Operations, NDIA 
¢ Mr. Bob Rassa, Director, Systems Support, 
Raytheon Company 
8:15 AM - 9:00 AM Keynote Grand Mesa DEF 
¢ VADM Walter B. Massenburg, USN, 
Commander, Naval Air Systems Command 
9:00 AM - 9:45 AM CMMI® Into the Future Grand Mesa DEF 
¢ Mr. Clyde Chittister, Chief Operating Office, SEI 
¢ Mr. Bob Rassa, Director, Systems Support, 
Raytheon Company 
9:45 AM - 10:15 AM Break Atrium Display Area 
10:15 AM - 11:45 AM Executive Panel Grand Mesa DEF 
¢ Moderator: Mr. Mark Schaeffer, Director, Systems & 
Software Engineering, OSD (AT&L) 
¢ Mr. Roger Fujii, Vice President, Communications 
and Systems Technology, Defense Mission 
Systems Division, Mission Systems Sector, 
Northrop Grumman Corporation 
¢ Mr. Dev Banerjee, Division Director, Systems & 
Flight Engineering, Integrated Defense Systems 
St. Louis, The Boeing Company 
12:00 PM - 1:30 PM Lunch with Guest Speaker Grand Mesa ABC Corridor 
¢ Mr. Mark Schaeffer, Director, Systems & 
Software Engineering, OSD (AT&L) 


1:30 PM - 5:00 PM Technical Sessions Refer to Following Pages 
3:00 PM - 3:30 PM Break Atrium Display Area 
5:00 PM - 6:30 PM Reception Atrium Display Area 


WEDNESDAY, NOVEMBER 15, 2006 


7:15 AM - 5:00 PM Conference Registration Open Grand Mesa Foyer 

7:15 AM - 8:15 AM Continental Breakfast Atrium Display Area 

8:15 AM - 11:45 AM Technical Sessions Refer to Following Pages 
9:45 AM - 10:15 AM Break Atrium Display Area 

12:00 PM - 1:30 PM Lunch Grand Mesa ABC Corridor 
1:30 PM - 5:00 PM Technical Sessions Refer to Following Pages 
3:00 PM - 3:30 PM Break Atrium Display Area 


THURSDAY, NOVEMBER 16, 2006 


7:15 AM - 11:45 AM Conference Registration Open Grand Mesa Foyer 

7:15 AM - 8:15 AM Continental Breakfast Atrium Display Aera 

8:15 AM - 11:45 AM Technical Sessions Refer to Following Pages 
9:45 AM - 10:15 AM Break Atrium Display Area 


11:45 AM Conference Adjourns 


Technical Sessions - Tuesday, November 14, 2006 
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Dr. Gene Miluk, SEI 


Corporation 


Improvement Using RUP 
Mr. John Miller, Cognence, Inc. 


High Maturity 
Dr. Richard 
Turner, Systems 
and Software 
Consortium, Inc. 


2D2 

Report to OSD: NDIA Top 5 
Soitware Issues 

Mr. Bob Rassa, Raytheon 
Company 


2D2 

Right Turns and Wrong Turns to 
Level 5 

Mr. Timothy Davis, Raytheon 
Company, Missile Systems 


Appraisals 
Mr. Geoff Draper, 
Harris Corporation 


2D3 

ISO and CMMI: How to Perform 
SCAMPI Appraisals and ISO Audits 
in a Syncronized, Simultaneous, 
Stupendous One Fell Swoop 

Ms. Margaret Glover, SEI 


2D3 

Experience Report - An ISO/IEC 
15504-conformant SCAMPI Appraisal 
Mr. David Kitson, SEI 


ROI & Benefits of 


2D4 
The Rosetta Stone CMMI ROI 


2D4 
Quick Win Strategies in CMMI 


CMMI P Model - Translating Between ROI, Transition 
Dr. Dennis CMMI and Metrics Ms. Angel (Qi) Liu, Motorola GSG, 
Goldenson, SEI Mr. Fionbarr McLoughlin, China 

ss University of Limerick 
CMMI Extensions: 2D5 ; 2D5 : 
Acquisition The Hidden Truths of Being a When CMMI-AM No Longer Suffices 


Mr. Jeffrey Dutton, 
Jacobs Technology 


Successful Acquirer 
Dr. Hubert Hofmann, General 
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Mr. Claude Marais, TPI, Inc. 


Systems 
Engineering 

Mr. Jerry Fisher, 
Aerospace 
Corporation 


2D6 

No Doubt: The LRAS3/FS3 Story of 
Mission Assurance 

Mrs. Jill Brooks, Raytheon 
Company 


Lean, Six Sigma 

and Agile 

Ms. Susan Bassham, 
US Army 


2D7 

Using Business Process 
Management Technology to 
Implement CMMI-compliant Agile 
Software Development 

Mr. Robert Moore, III, Business 
Transformation Institute 


2D7 

Using “Voice of Customer” Tools to 
Advance Organizational Innovation 
and Deployment 

Mr. Donald Corpron, Northrop 
Grumman Corporation 
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Technical Sessions - Wednesday, November 15, 2006 


Session/Chair 


Session A 


Session A 


Session/Chair 


Session B 


Session B 


8:15 AM 


9:00 AM 


10:15 AM 


11:00 AM 


A/d ese 
pueig 
T9edL 


CMMI and Process 
Improvement 
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CMMI-based Process Improvement 
in Motorola? 

Ms. Angel (Qi) Liu, Motorola GSG, 
China 
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Learning to Count - Do You 
Understand Why Level 3 Comes 
Before Level 4 Which Comes Before 
Level 5? 

Ms. Nancy Raymond, Raytheon 
Corporation 
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Appraisals 
Mr. Geoff Draper, 
Harris Corporation 
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Passing an Appraisal Without 
Preparation 

Mr. Kent McClurg, Raytheon 
Company 
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CMMI and Process 
Improvement 
Dr. Gene Miluk, SEI 


3B1 

Organizational Synthesis: CMMI- 
The Glue that Binds 

ey ea Arnold, BAE Systems, 


3B1 

Operations and Maintenance - More 
Than Just a Scaling Issue 

Mr. Laurence Gill, BAE Systems 


High Maturity 
Mr. Joe Elm, SEI 


3B2 

Making OID Effective 

Ms. Diane Mizukami Williams, 
Northrop Grumman Corporation, 
Mission Systems 


3B2 

When OID Crosses Time Zones, 
Disciplines and Customer Base 
Ms. Laurie Haack, Raytheon 
Company 


Appraisals 
Mr. Geoff Draper, 
Harris Corporation 


3B3 

Lessons Learned from Conducting 
Scampi B and C Appraisals 

Mr. Itzhak Lavi, Israel Aircraft 
Industries, Ltd. 


3B3 

Practical Advice on Picking the Right 
Projects for an Appraisal 

Mr. Robert Moore, III, Business 
Transformation Institute 


Measurement 
Dr. Dennis 
Goldenson, SEI 


3B4 

Practical Earned Value Management 
Integrated with CMMI-compliant 
Processes 

Dr. Donn Milton, Pragma Systems 
Corporation 


3B4 

extant Suede Value from CMMI 
Dr. Rick Hefner, Northrop 
Grumman Corporation 


CMMI Extensions: 
Acquisition 

Mr. Jeffrey Dutton, 
Jacobs Technology 


3B5 

SCAMPI B's for Contract Monitoring 
- A Case Study of the Mission 
Planning Enterprise Contractors 
Ms. Lorraine Adams, SEI 


3B5 

DCMA CMMlI-related Data Collection: 
A Maturity Model-based Reflection of 
Practice Strengths and Weaknesses 

Mr. John Eget, DCMA 
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Systems Engineering Revitalization 
Using CCMI 

Mr. Michael Kutch, Jr., SPAWAR, 
Charleston 


Lean, Six Sigma 

and Agile 

Ms. Susan Bassham, 
US Army 


3B7 

Improving Software Development: 
Starting & Sustaining Lean & CMMI 
in Federal Agencies 

Dr. Mel Adams, University of 
Alabama, Huntsville 
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Using Design for Six Sigma (DFSS) 
to Improve the Ease-of-use of a 
Process Asset Library Seay 

Mr. Fred Roberts, Genera 
Dynamics, Advanced Information 
Systems 
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Technical Sessions - Wednesday, November 15, 2006 
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Session C 


Session/Chair 


Session D 


Session D 


1:30 PM 


2:15 PM 


3:30 PM 


4:15 PM 


3/0 esa 
pues 
Toe 


CMMI and Process 
Improvement 
Dr. Gene Miluk, SEI 


3C1 
Program Management Office (PMO) 
and CMMI - Paving the Road to 
Optimized IT Value: Case Study 

Ir. Terry Jabali, Applied 
Enterprise Dynamics, Inc. 


3C1 

How to Sell CMMI to Your CEO in His 
Own ee 
Ms. Lemis Altan, Cognence, Inc. 
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High Maturity 
Mr. Joe Elm, SEI 


3C2 

Looking Beyond Quantitative Defect 
Management 

Mr. Anil Midha, BAE Systems 


3C2 

Defect Analysis - The Devil is in the 
Details, or Lack Therof 

Mr. Tobias Cahoon, 96th 
Communications Group, USAF 
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Appraisals 
Mr. Geoff Draper, 
Harris Corporation 


3C3 

Lessons Learned: Getting Ready for 
a SEI Maturity Level 5 

Mr. Dennis Ring, Lockheed Martin 
Corporation, IS&S 


3C3 
Indicators of Reliable SCAMPI 
Appraisal Results 

ir. Bruce Boyd, The Boeing 
Company 
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Measurement 
Dr. Dennis 
Goldenson, SEI 


3C4 

Performance Benchmarking 
Consortium 

Mr. Mark Kasunic, SEI 


= CMMI Extensions: 3C5_ 3C5 
oO Acquisition Practical Use of DCMA’s CMMI- The Issue of Performance - Wh 
oo based Risk Management Do You Need a Maturity Level 
» » Mr. Jeffrey Dutton, Methodology Supplier? 
<> Jacobs Technology Mr. Gary Sigmund, DCMA Mr. Tim Kasse, Kasse Initiatives, 
2 ge LLC 
joe 
oO 
= eer CMMI V1.2 Overvi SCAMPI 1,2 Updat 
s .2 Overview .2 Updates 
Sa Beyond , Mr. Mike Phillips, SEI Dr. Jack Ferguson, SEI 
Q 2 Mr. Hal Wilson, 
Ax Northrop Grumman 
S a Corporation 
2 
< Lean, Six Sigma ee ss pesianeke rae 
i ean Process Design - A Concept o' 
=a and Agile Process Quality 
a s Ms. Susan Bassham, Mr. David Card, Q-Labs 
= US Army 
on 
is} 
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CMMI and Process 
Improvement 
Dr. Gene Miluk, SEI 


3D1 

Make Middle Managers the Process 
Owners 

Mr. Tim Kasse, Kasse Initiatives, 
LLC 


3D1 

You Shouldn’t Have to Be an 
Engineer to Understand Your 
Process 

Ms. Leticia Santillan, Raytheon 
Company, Missile Systems 


High Maturity/ 
Sustaining 
Improvement 
Mr. Joe Elm, SEI 


3D2 

Statistical Control of System and 
Software Design Activities 

Dr. Richard Welch, Northrup 
Grumman Corporation 


3D2 

| Have Had My CMMI Appraisal... 
What Do | Do Now? How to Establish 
a Process Improvement Work 
Breakdown Structure 

Dr. Aldo Dagnino, SAS Institute 


Implementation 
Guidance 

Mr. Geoff Draper, 
Harris Corporation 


3D3 

Project Implementation Strategies in 
the CMMI 

Dr. Rick Hefner, Northrop 
Grumman Corporation 


3D3 

Patterns: An Approach for CMMI 
Adoption 

Dr. Rick Hefner, Northrop 
Grumman Corporation 


Measurement 
Dr. Dennis 
Goldenson, SEI 


3D4 

2006 State of Software Measurement 
Practice Survey 

Mr. Mark Kasunic, SEI 


3D4 

Measures and Lessons from a CMMI 
Guided Process Improvement Project 
ag oe Buys, 3H Technology, 


CMMI Extensions: 
Services 

Mr. Jeffrey Dutton, 
Jacobs Technology 


3D5 

CMMI for Services: Introducing the 
CMMI for Services Constellation 
Mr. Craig Hollenbach, Northrop 
Grumman Corporation 


3D5 

CMMI Implementation in 
Manufacturing Processes 

Mr. Itzhak Lavi, Israel Aircraft 
Industries, Ltd. 


CMMI v1.2 and 
Beyond 

Mr. Hal Wilson, 
Northrop Grumman 
Corporation 


3D6 

Strengthening the Credibility of 
Level 4/5 Appraisals 

Mr. Will Hayes, SEI 


3D6 
Lead Appraiser Training for V1.2 
Dr. Jack Ferguson, SEI 


CMMI in Small 
Settings 

Ms. Susan Bassham, 
US Army 


3D7 

Easing Project Planning for Small 
Programs 

Mr. Kenneth Weinberg, Raytheon 
Company 


3D7 

The Road to Lean CMMI in a Small 
Federal Organization 

Ms. Susan Bassham, US Army, 
Aviation and Missile Command 
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Technical Sessions - Thursday, November 16, 2006 
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Session/Chair Session A Session A Session/Chair Session B Session B 
8:15 AM 9:00 AM 10:15 AM 11:00 AM 
poe er eess Movi from Level 2 to Level 3 Risk ined dvs. Integrated P oir pecoh ila Using Best Practices at NASA to 
joving from Level 2 to Level 3 Risk _ Interfaced vs. Integrated Process r i 
improvement Management Mr. Robert Rice, BAE Systems Improvement Define a CMMI DAR Process that 


Dr. Gene Miluk, SEI 


poe Susan Byrnes, Natural SPI, 
Inc. 


nine Mot ing the T Conti | ing f the M in - Staying A 
jotivating the Team to Continuously King of the Mountain - Staying Atop 

<= aad Improvement Improve After the CMMI Rating Mt. CMMI After the Initial Assent 
O-7 Ms. Lorraine Ms. Leticia Santillan, Raytheon Mr. Timothy Davis, Raytheon 
$ 2 2. Adams, SEI Company, Missile Systems Company, Missile Systems 
T1°2%n 

Implementation 4A3 , 4A3 : 

Guidance An Enterprise Approach to Process += CMMI and ISO 9001- A Registrar's 

Improvement Perspective 
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Mr. Geoff Draper, 
Harris Corporation 


Ms. Linda Kovar, Raytheon 
Company 


Mr. Matt Leiphart, Platinum 
Registration 
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Measurement 
Dr. Dennis 
Goldenson, SEI 


4A4 

Integrating Planning, Control, 
Measurement & Analysis CMMI 
Process Areas to Achieve More 
Successful Software Projects 
Mr. Daniel Galorath, Galorath 
Incorporated 


4A4 

The Unseen Benefits of the Change 
Request Process 

Mr. Scott Banks, Georgia Tech 
Research Institute 


= CMMI Extensions: ie ficeersinetd a fs aa Havel fl 
i, essons in Integrating Non- ngineering in a Non-developmen 

® = Services engineering Organizations with Environment 

2 » Mr. Jeffrey Dutton, — Engineering Using the CMMI Ms. Kathy King, BAE Systems 

<2 Jacobs Technology r. Joseph Vandeville, Northrop 

DO & Grumman Corporation 

rey 

fo) 

= ener Implications of Combining SAM and A Practical Roadmap f 

= mplications of Combining an ractical Roadmap for 

So Beyond — ISM in CMMI V1.2 Transitioning to CMMI V1.2 

ag Mr. Hal Wilson, Mr. Mike Phillips, SEI Dr. Rick Hefner, Northrop 

Dx Northrop Grumman Grumman Corporation 

a Corporation 

= 

= ee CMMI A lications in Small Setti ire Il Project Chall f CMMI i 
; pplications in Small Setting mall Project Challenges o' in 

i | Settings Finance Gioaneotan for Process Large Organizations 

a8 Ms. Susan Bassham, Performance Optimization Mr. John Bethmann, Concurrent 

yn * US Army Dr. Mary Anne Herndon, Technologies Corporation 

a ~N Transdyne Corporation 
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Dr. Gene Miluk, SEI 


Addresses Dissent 
Mr. Tim Olson, Qualit, 
Improvement Consultants, Inc. 


Sustaining ae ee 4B2 ; 
ustaining ompliance Maintaining Institutionalization After 

Imp rovement Dr. Rick Hefner, , the Appraisal Team Leaves 

Ms. Lorraine Northrop Grumman Corporation — Mr. John Johnston, PMP, 

Adams, SEI BAE Systems 

Implementation 4B3 4B3 ; a 

Guidance It's a #$*!>% Mess: How to Leverage Systematically Tailoring the 


Mr. Geoff Draper, 


Today's Process into Tomorrow's 
Initiatives 


Organization’s Standard Process for 


a Project 


i i Ms.Terri Bryant, Raytheon Dr. Kenneth Nidiffer, Systems and 
Harris Corporation Company, Missile Systems DR Cope ceucie 
Measurement 4B4 4B4 _ 

Dr. Dennis Measuring the Impact of RD and Untethered Activities - The Real 


Goldenson, SEI 


REQM CMMI Process Areas 
Dr. Aldo Dagnino, SAS Institute 


Reason for Schedule Slips 
Mr. Rolf Reitzig, Cognence, Inc. 


CMMI Extensions: 
Services 

Mr. Jeffrey Dutton, 
Jacobs Technology 


4B5 

Achieving Delivery Excellence Using 
CMMI 

Mr. Keith Heston, Accenture 


CMMI v1.2 and re ina RR 
esolving ssues 
Beyond “ Dr. Karen Richter, Institute for 
Mr. Hal Wilson, Defense Analyses 
Northrop Grumman 
Corporation 
CMMI and 4B7 
Standards any eee Between CMMI, 


Ms. SusanBassham, 
US Army 


, TIL and Six aiinae 
Why Not Leverage All! 

ue Lemis Altan, Cognence, 
Inc. 
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Tutorial Sessions - Monday, November 13, 2006 


Session A 


Session B 


Session C 


Session D 


8:00 AM 


10:15 AM 


1:00 PM 


3:15 PM 
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1A1 Tutorial 
CMMI v1.2 Tutorial 
Mr. Mike Phillips, SEI 


4 esa 
pueis 


2 YOeAL 


1A2 Tutorial 

The Return on Investment from CMMI and 
Other Best Practices: An Introduction 

Mr. Thomas McGibbon, ITT AES 


spue|yblyq 


€ YOeAL 


1A3 Tutorial 

Addressing ay ee Project 
Management Activities Using Process 
Simulation & Supporting C’ 
Areas 

Dr. David Raffo, Portland State 
University 


| for Process 


V 991D 
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1A4 Tutorial 

Integrated Project Management (IPM) - 
The CMMI and Collaborative Product 
Development 

Mr. William Deibler, Il, Software 
Systems Quality Consulting 
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1A5 Tutorial 

A Practical Guide to Implementing Levels 
4and5 

Dr. Rick Hefner, Northrop Grumman 
Corporation 
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1A6 Tutorial 

How to Measurably Improve Your 
Requirements 

Mr. Tim Olson, Quality Improvement 
Consultants, Inc. 
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1A7 Tutorial 

Lessons Learned in Implementing 
CMMI in a Large Systems Engineering 
Organization 

Mr. Michael Kutch, Jr., SPAWAR, 
Charleston 
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1A8 Tutorial 
Lean Process Improvement 
Mr. Jeffrey Dutton, Jacobs Technology 
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1B1 Tutorial 
CMMI v1.2 Tutorial (continued) 
Mr. Mike Phillips, SEI 


1B2 Tutorial 
The Return on Investment from CMMI and 
Other Best Practices: An Introduction 
continued) 

Ir. Thomas McGibbon, ITT AES 


1B3 Tutorial 

Addressing High-Value Project 
Management Activities Using Process 
Simulation & Supporting CMMI for Process 
Areas (continued) 

Dr. David Raffo, Portland State 
University 


1B4 Tutorial 

Integrated Project Management (IPM) - 
The CMMI and Collaborative Product 
Development (continued 

Mr. William Deibler, Il, Software 
Systems Quality Consulting 


1B5 Tutorial 

A Practical Guide to Implementing Levels 
4and5 (omen) 

Dr. Rick Hefner, Northrop Grumman 
Corporation 


1B6 Tutorial 

How to Measurably Improve Your 
Requirements (continued) 

Mr. Tim Olson, Quality Improvement 
Consultants, Inc. 


1B7 Tutorial 

Lessons Learned in Implementing 
CMMI in a Large Systems Engineering 
Organization (continued) 

Mr. Michael Kutch, Jr., SPAWAR, 
Charleston 


1B8 Tutorial 
Lean Process Improvement (continued) 
Mr. Jeffrey Dutton, Jacobs Technology 
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1C1 Tutorial j 
Are We Ready for CMMI? If Not, Let’s Fix 
Ourselves 


Mr. Al Florence, The MITRE Corporation 


1C2 Tutorial 

Calculating CMMI-based ROI: How, What, 
When and Why? 

Mr. Rolf Reitzig, Cognence, Inc. 


1C3 Tutorial 

Into the Future, Part 1: Process Definition 
on Steroids 

Ms. Kathy Reid, 96th Communications 
Group/TYBRIN, USAF 


1C4 Tutorial 

CMMI Crash Course: What the SEI Won't 
Teach You 

Mr. Hillel Glazer, Entinex, Inc. 


1C5 Tutorial 

Basics of PSP and TSP for Systems 
Engineering 

Mr. James McHale, SEI 


1C6 Tutorial 

Requirements Engineering: A Practical 
Approach to Modeling and Managing 
Requirements 

Mr. William Deibler, Il, Software 


Systems Quality Consulting 


1C7 Tutorial 

ISO/IES 15288 and CMMI(SM): Systems 
Engineering Similarities & Differences 
Mr. David Walden, Sysnovation, LLC 


1C8 Tutorial 

How to Define Lean Processes that are 
CMMI-compliant 

Mr. Tim Olson, Quality Improvement 


Consultants, Inc. 
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1D1 Tutorial : 
Are We Ready for CMMI? If Not, Let’s Fix 
Ourselves (continued) . 
Mr. Al Florence, The MITRE Corporation 


1D2 Tutorial 

Calculating CMMI-based ROI: How, What, 
When and Why? (continued) 

Mr. Rolf Reitzig, Cognence, Inc. 


1D3 Tutorial 

Into the Future, Part 1: Process Definition on 
Steroids (continued aoe 

Ms. ay Reid, 96th Communications 
Group/TYBRIN, USAF 


1D4 Tutorial 

CMMI Crash Course: What the SEI Won't 
Teach You (continued) 

Mr. Hillel Glazer, Entinex, Inc. 


1D5 Tutorial 

Basics of PSP and TSP for Systems 
Engineering eomtnuet) 

Mr. James McHale, SEI 


1D6 Tutorial : 
Requirements Srgneeting, A Practical 
Approach to Modeling and Managing 
Requirements (continued) 

Mr. William Deibler, II, Software 
Systems Quality Consulting 


1D7 Tutorial 
ISO/IES 15288 and CMMI(SM): Systems 
Engineering Similarities & Differences 
continued) 

Ir. David Walden, Sysnovation, LLC 


1D8 Tutorial 

How to Define Lean Processes that are 
CMMI-compliant (continued) 

Mr. Tim Olson, Quality Improvement 
Consultants, Inc. 
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Carnegie Mellon 
Software Engineering Institute 


Pittsburgh, PA 15213-3890 


CMMI® Version 1.2 and Beyond 


NDIA CMMI Technology Conference 
November 13, 2006 


Mike Phillips 
Software Engineering Institute 
Carnegie Mellon University 


® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. 


Thanks to Gary Wolf and D’Ann Hunt from Raytheon, Denise Cattan of SPIRULA 
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Carnegie Mellon 
Software Engineering Institute 


Pittsburgh, PA 15213-3890 


CMMI Today 


page 2 


—=— Carnegie Mellon 
—— Software Engineering Institute 


Pittsburgh, PA 15213-3890 


Selected CMMI® Data (FY06) 


17,630 Trained in Introduction to CMMI 


Net increase of: 
-108 Authorized Introduction to CMMI Instructors 
= 45 Authorized SCAMPI Lead Appraisers 
= 84 Authorized SCAMPI B/C Team Leads 


744 SCAMPI V1.1 Class A Appraisals conducted and 
reported to the SEI 


417 Publicly Reported SCAMPI V1.1 Class A 
Appraisals on the SEI Web Site 
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=— Carnegie Mellon 
—— Software Engineering Institute 


Number of CMMI Students Trained 
(Cumulative) 


30000 

25000 

20000 

15000 = CMMI (Staged) discont'd. 
12/31/05 

10000 & CMMI (Continuous) 
discont'd. 12/31/05 

5000 B CMMI (S&C Combined) 
0 


1999 2000 2001 2002 2003 2004 2005 YTD 
2006 


8-31-06 


=— Carnegie Mellon 
—— Software Engineering Institute 


Transition from V1.1 to V1.2 Status - 10-12-06 


Introduction to CMMI Students 
e Registered for Upgrade Training — 464 
e Upgrades Complete — 190 


Lead Appraisers and Instructors 
e Registered for Upgrade Training — 561 


¢ Upgrade Training Complete — 256 
e Exams 
- Passed — 159 


- Failed (and not yet retaken) -— 7 


=— Carnegie Mellon 
—— Software Engineering Institute 


CMMI Adoption Trends 


CMMI Web pages views in 
September 2006 

¢ 360K/month 

e Exceeded 24K/day 
The following were the most 
downloaded files on the SEI 


Web site in September 2006: 


¢ CMMI-DEV, V1.2 (PDF 
Version) 

¢ CMMI V1.1 Overview 
Presentation 

¢ CMMI V1.2 Overview 
Presentation 

¢ CMMI-DEV, V1.2 (Word 
Version) 


¢ CMMI V1.2 Model Changes 


Presentation 


© 2006 by Carnegie Mellon University 


Average daily page views per quarter 


Carnegie Mellon 
Software Engineering Institute 


CMMI Transition Status — 10/31/06 
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Training 
Introduction to CMMI — 58,178 trained 
Intermediate CMMI — 2,280 trained 
Introduction to CMMI Instructors — 449 
SCAMPI Lead Appraisers — 630 trained 
SCAMPI B&C-Only Team Lead -- 33 


Authorized 
Introduction to CMMI V1.1 Instructors — 404 
SCAMPI V1.1 Lead Appraisers — 451 
SCAMPI B&C Team Leads -- 432 


—— Carnegie Mellon 


~—— Software Engineering Institute 
Intro to the CMM and CMMI Attendees 
(Cumulative) 
60000 
50000 
40000 
© CMM Intro (discon'td. 
30000 12/31/05) 
@ CMMI Intro 
20000 
10000 B@ CMMI Intennediate 
0 


— eo 1997 make 1999 2000 2001 2002 2003 2004 2005 YTD 7-31-06 
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Carnegie Mellon 
Software Enaineerina Institute 


Number of Appraisals Conducted by Year 
Reported as of 31 August 2006 


1000 


900 


800 


700 


600 


500 


400 


o 


1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 


G@SPA @ CBAIPI(Discontinued after 12/31/2005) O SCAMPI vX ClassA 


Sis C . M 1] 
= Software Engineering Institute 


Number of SCAMPI vX Class A Appraisals Conducted by Year by Model 
Representation* 
Reported as of 31 August 2006 


*Where Representation is reported 


@ Staged 


® Continuous 
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Number of SCAMPI v1.1 Class A Appraisals Conducted by Quarter 
Reported as of 31 August 2006 
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Q2/02 Q3/02 Q4/02 Q1/03 Q2/03 Q3/03 Q4/03 Q1/04 Q2/04 Q3/04 Q4/04 Q1/05 Q2/05 Q3/05 Q4/05 Q1/06 Q2/06 


Carnegie Mellon 
Software Engineering Institute 


Reporting Organizational 
Categories 


Commercial/in-house 


67.6% 


Contractor for 
Military/Government 
28.8% 


Military/Government 
Agency 


0 200 400 600 800 1000 1200 
Number of Organizations 


Based on1,377 organizations 


= Carnegie Mellon 
Software Engineering Institute 


Organization Size 
Based on the total number of employees within the area of the organization 


—— 
——- 
_— 


that was appraised 
1001 to 2000 
a 2000+ 
501 to 1000 — | 2.8% 
8.2% / 25 or fewer 
nel 9.8% 
301 to 500 26 to 50 
95% ~ 13.4% 
eee pa 1 to 100 
: 43.0% 
a _ 51to 75 
201to300 ree 
10.9% 101 to 200 
20.0% 
\. 76 to 100 
8.7% 


Based 0n1,348 organizations reporting size data 


Carnegie Mellon University 
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Maturity Profile by All Reporting 
USA and Non-USA Organizations 


100% 
90% + @ USA: 100 % = 498 
80% —- @ Non-USA: 100 % = 879 


60% -- 


40% -—- 188 316 


Organizations 
o1 
S 
& 


% of 


Not Given Initial Managed Defined Quantitatively Optimizing 
Managed 


Based 0n498 USA organizations and879 Non-USA organizations 


Carnegie Mellon 
Software Engineering Institute 


Publicly Reported SCAMPI V1.1 Class A Appraisals 
— 9/30/06 


Distribution of Publicly Reported Appraisals 


Lev 2 Lev 3 Lev 4 Lev 5 Not 
Assigned 


Carnegie Mellon 


COtinttiés Where Appraisals have 
been Performed and Reported to 


Argentina Australia Austria Bahrain Belarus Belgium Brazil Canada 

Chile China Colombia Czech RepublicDenmark Dominican Republic Egypt Finland 

France Germany Hong Kong India Indonesia Ireland Israel __ Italy 

Japan Korea, Republic of Latvia Malaysia Mauritius Mexico Morocco Netherlands 
New Zealand Pakistan Philippines Portugal Russia Singapore Slovakia South Africa 
Spain Sweden Switzerland Taiwan Thailand Turkey Ukraine United Kingdom 


United States Vietnam 


Red country name: New additions with this reporting 


| 


Carnegie Mellon 
Software Engineering Institute 


Number of Appraisals and Maturity 
Levels Reported to the SEI by Countr 


i 


Maturity Maturity Maturity Maturity Maturity Maturity Maturity Maturity Maturity Maturity 


Number of Level1 Level2 Level3 Level4  Level5 Number of Level1 Level2 Level3 Level4  Level5 
Appraisals Reported Reported Reported Reported Reported Country Appraisals Reported Reported Reported Reported Reported 


Argentina iy No Yes Yes Yes Yes Korea, Republic of56 Yes Yes Yes Yes Yes 
Australia "23 Yes Yes Yes Yes Yes Latvia 10 or fewer 
Austria 10 or fewer Malaysia "15 No No Yes No Yes 
Bahrain 10 or fewer Mauritius 10 or fewer 
Belarus 10 or fewer Mexico 10 or fewer 
Belgium 10 or fewer Morocco 10 or fewer 
Brazil 39 No Yes Yes Yes Yes Netherlands 10 or fewer 
‘Canada "18 No Yes Yes No Yes New Zealand 10 or fewer 
Chile 10 or fewer Pakistan 10 or fewer 
‘China "158 Yes Yes Yes Yes Yes Philippines "4 No Yes Yes No Yes 
Colombia 10 or fewer Portugal 10 or fewer 
Czech Republic 10 or fewer Russia 10 or fewer 
Denmark 10 or fewer Singapore 10 or fewer 
Dominican Republic 10 or fewer Slovakia 10 or fewer 
Egypt "10 South Africa 10 or fewer 
Finland 10 or fewer Spain 25 No Yes Yes No Yes 
France 65 Yes Yes Yes Yes Yes Sweden 10 or fewer 
Germany "28 Yes Yes Yes Yes Yes Switzerland 10 or fewer 
Hong Kong 10 or fewer Taiwan 21 No Yes Yes No No 
India "TT No Yes Yes Yes Yes Thailand 10 or fewer 
Indonesia 10 or fewer Turkey 10 or fewer 
Ireland 10 or fewer Ukraine 10 or fewer 
Israel 10 or fewer United Kingdom 42 Yes Yes Yes Yes No 
Italy 10 or fewer United States 598 Yes Yes Yes Yes Yes 
Japan "155 Yes Yes Yes Yes Yes Vietnam 10 or fewer 
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Number of 


Appraisals 


Carnegie Mellon 


Software Engineering Institute 


Disciplines Selected for 


) Appraisals 


900 


50.7% 


800 


700 


600 36.2% 


500 


400 


300 


Based 0n1,581appraisals 


For more information about Allowable Models & Combinations, visit: http:/Awww.sei.cmu.edu/cmmi/background/aspec.html 


3.9% 


SSIMSIAS 


3.5% 


SS/Oddl/IMS/IAS 


0.9% 


Odd lIMS 


SE = System Engineering 
SW _ = Software Engineering 
IPPD = Integrated Product and 


Process Development 
SS = Supplier Sourcing 


-2% 0.1% 


SS/Oddl/AS 


0.1% 


SS/OddlIMS 


=— Carnegie Mellon 
~—— Software Engineering Institute 


Combined Appraisal Opportunities 


Current ISO 9001 (Combined ISO Surveillance 
SCAMPI ‘A’ using Cat ‘C’ appraisal) 
ISO 9001 z = 


IA 
ISO 9001 


... continues to 
demonstrate 
compliance with 
ISO 9001:2000 


Rating letter & or certificate 
with scope indicating 
“,,. Inaccordance with Level X” 


...no behaviours 
inconsistent with 
operating at level X 


Rating letter 
indicating level 
achieved 


Current CMMI 


(Cat ‘C’ appraisal) 


ane possible peat for assessment and euyeINeaneS: 
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—— Carnegie Mellon 


—— Software Engineering Institute 
Adoption: What Else Is Happening? 


The Addison-Wesley SEI Series Book and: 
e A Guide to the CMMI 

CMMI: A Framework... 

CMMI Assessments 

CMMI Distilled: Second Edition 

CMMI SCAMPI Distilled 

CMMI: Un Itinéraire Fleche 

De kleine CMMI 

Interpreting the CMMI 

Making Process Improvement Work 

Practical Insight into CMMI 

CMMI Survival Guide 

Real Process Improvement Using the CMMI 


Systematic Process Improvement Using ISO 
9001:2000 and CMMI 


Balancing Agility and Discipline 
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Carnegie Mellon 
Software Engineering Institute 


How about SEI Publications? 


Technical notes and special reports: 

e Interpreting CMMI: 
- for Operational Organizations 
- for COTS Based Systems 
- for Service Organizations 
- for Marketing (in progress) 

¢ Using CMMI with: 
- Earned Value Management 
- Product Line Practices 
- Six Sigma 

¢ Supplementing CMMI for Safety Critical Development 
(in progress) 

e Demonstrating the Impact and Benefits of CMMI (and 
web pages — www.sei.cmu.edu/cmmi/results) 
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Performance Results Summar 


# of 
data 
Improvements Median | points Low High 


Cusiomer 14% 7 -A% | 55% 
Satisfaction 

Return on 40:1 99 17:1 21.1 * 
Investment 1 


¢ N= 30, as of August 2006 
¢ Organizations with results expressed as change over time 
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CMMI Today 


Version 1.1 CMMI Product Suite was released January 
2002. 


¢ CMMI Web site visits average over 12,000/day 
¢ Over 58,000 people have been trained 


¢ Almost 1600 “class A” appraisals have been 
reported to the SEl 


Now we want to continuously improve... 
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CMMI V1.2...and Beyond 
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Major Themes 


Reduce complexity & size 
Increase coverage 


Increase confidence in appraisal results 
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Reduced Model Complexity & Size 


Eliminated the concepts of advanced practices and 
common features 


Incorporated ISM into SAM; eliminated Supplier Sourcing 
(SS) addition 


Consolidated and simplified the IPPD material 
All definitions consolidated in the glossary 


Adopted a single book approach (i.e., will no longer 
provide separate development models) 


Report size reduced 15% from either predecessor; PAs 
reduced 12% 
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Increased Model Coverage 


Added hardware amplifications 


Added two work environment practices (i.e., one in OPD 
and one in IPM) 


Added goal and two practices in OPF to emphasize 
importance of project startup 


Updated notes (including examples) where appropriate so 
that they also address service development and 
acquisition of critical elements 


Updated name to CMMI for Development (CMMI-DEV) to 
reflect the expanded coverage 
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Model Changes - Other 


Improved the Overview section (Part One) 
Improved clarity of how GPs are used 
e Moved generic goals and practices to Part Two 


e Added explanation of how process areas support the 
implementation of GPs 


e Added GP elaborations for GP 3.2 


Improved the glossary (e.g., higher level management, 
bidirectional traceability, subprocess) 


Limited the process areas that can be considered “not 
applicable” to SAM. 


Clarified material throughout the model based on over 1000 
change requests 
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Integrated Product and Process 
Development (IPPD) Changes 


IPPD material is being revised significantly. 


¢ Organization Environment for Integration PA removed 
and material moved to Organizational Process 
Definition (OPD) PA. 


e Integrated Teaming PA removed and material moved to 
Integrated Project Management (IPM) PA. 


e IPPD goals have been consolidated. 
- “Enable IPPD Management” in OPD 


- “Apply IPPD Principles” in IPM 


¢ Overall material condensed and revised to be more 
consistent with other PAs. 
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CMMI Model Combinations 
V1.1 V 1.2 


. 
Organizational Goal 
(OPD) 
Project Goal (IPM) 
Le 
Related 


Examples 


——_—— ap 
Supplier 
Sourcing 


ntegrated Product anc 
Process Development 
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IPPD Changes 


V1.1 
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——_—O  siirp principles | 


SG3 = Apply 
IPPD principles 
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Supplier Agreement Management 


Specific Goal Specific Practice 
Establish Supplier 1.1 - Determine Acquisition Type 
Agreements 1.2 - Select Suppliers 


1.3 - Establish Supplier Agreements 


Satisfy Supplier 2.1 — Execute the Supplier Agreement 
Agreements 2.2 — Monitor Selected Supplier 
Processes 
2.3 — Evaluate Selected Supplier Work 
Products 


2.4 — Accept the Acquired Product 
2.5 — Transition Products 


v1.1 SP2.1 “Review COTS Products,” was eliminated. “Identify 
candidate COTS products that satisfy requirements” is a new 
subpractice under the Technical Solutions Process Area SP1.1, 
“Develop Alternative Solutions and Selection Criteria.” 


© 2006 by Carnegie Mellon University page 32 


=— Carnegie Mellon 
-——— Software Engineering Institute 


Organizational Process Focus 


V1.1 V1.2 
SG1 - Determine Process Improvement SG1 - Determine Process Improvement 
Opportunities Opportunities 


1.1 - Establish Organizational Process Needs 1.1 - Establish Organizational Process Needs 
1.2 - Appraise the Organization’s Processes 1.2 - Appraise the Organization’s Processes 


1.3 — Identify the Organization’s Process 1.3 — Identify the Organization’s Process 
Improvements Improvements 

SG2 - Plan and Implement Process SG2 - Plan and Implement Process Improvement 
Improvement Activities 2.1 - Establish Process Action Plans 

2.1 — Establish Process Action Plans 2.2 — Implement Process Action Plans 
2.2 — Implement Process Action Plans SG3 - Deploy Organizational Process Assets 

2.3 —- Deploy Organizational Process Assets and Incorporate Lessons Learned 

2.4 - Incorporate Process-Related Experiences 3.1 — Deploy Organizational Process Assets 

into the Organizational Process Assets 3.2 -- Deploy Standard Processes 


3.3 -- Monitor Implementation 


3.4 -- Incorporate Process Related Experiences 
into the Organizational Process Assets 
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SCAMPI A Changes for V1.2 


Method implementation clarifications 
e interviews in “virtual” organizations 
° practice characterization rules 
¢ organizational unit sampling options 


Appraisal Disclosure Statement (ADS) improvements 
e reduce redundancy with other appraisal documents 
¢ improve usability for soonsor and government 
¢ Level 4,5 mapping to business objectives 
° require sponsor’s signature on the ADS 
e require all team members to show agreement on findings 
¢e Both V1,1 and V1.2 ADS reflect these today 


Appraisal team will have responsibility for determination of 
“applicability” for SAM 


Maturity level and capability level shelf life — 3 years, given 1 year 
of V1.2 availability 


Carnegie Mellon 
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Published Appraisal Results 


List of Published SCAMPI Appraisal Results 


ORGANIZATION NAME: Satyam Computer Services Ltd. 
SPONSOR NAME: Nagaraj Chevour 
LEAD APPRAISER NAME: Raghavan Nandyal 
SEI PARTNER: SITARA Technologies Pvt. Ltd. 
APPRAISAL END DATE: 4/3/2004 
MATURITY LEVEL ASSIGNED: 5 
APPRAISED ORGANIZATIONAL UNIT: 

Entity Name: SRU GE-GDC 

Location(s): Secunderabad, AP, India 
CMMI MODEL USED: CMMI-SYW/IPPD, V1.1, Continuous 
APPRAISAL METHOD USED: SCAMPI v1.1 


MODEL SCOPE & CAPABILITY RATINGS ASSIGNED: 
Process Management Project Management Engineering Support 


OPF (Capabiilevels) PP |Capabiivilevel4) REQM (Capabiilevels) cm (Capabilij/levels) 
OPD (CapabiliiLevels) PMC (CapebiliLeVel4) RD PPQA Capability Level 3 
OT (Capebiievels) SAM [ENBOABBIEZIEN 1S MA Capability Level 3. 
OPP (GapabiigLevels| IPM (CapSbinLEVeIS! PI DAR  Capabilty Level 3 
OID (CapebiiLEVelS)  § RSKM [CapabiivLeVel4) VER [Capabiiilevel5) OF |Capabilijilevels) 
IT — (CapabiIgILEVENS) «= VAL [CapabilifLeVelS) © CAR |CapabilifiLevels) 
ISM —sNotRated 
QPM Capatilty Level 3 
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CMMI Training v1.2 


Introduction to CMMI (Staged and Continuous) 
¢ Editorial update released 9/05 
e Updated v1.2 phased in this fall 


Intermediate Concepts of CMMI 
¢ Being updated for v1.2 
¢ will better prepare students for SCAMPI training 


CMMI Instructor Training 
¢ Being updated to reflect v1.2 changes 


“Delta” training from V1.1 to V1.2 
¢e Will be available on-line for free 
¢ More extensive upgrade course for fee 
- Appraisal Team members 
- Lead Appraisers, Instructors, candidates 
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CMMI V1.2 Schedules 


Version 1.2 CMMI Product 
Suite release 


Update material available 


August 25, 2006 


August 25, 2006 


Upgrade course available on- August 25, 2006 


line 


First Lead Appraiser “face to October 16, 2006 


face” 


V1.2 ADS required for all 
SCAMPIs 


Last V1.1 Intro training 


First expiration of V1.1 
appraisals (3 year validity) 


Last V1.1 appraisal 


October, 2006 


December, 2006 
August 25, 2007 


August 31, 2007 
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Beyond V1.2, 


Improved architecture will allow post-V1.2 expansion. 


e Extensions of the life cycle (Services, 
Outsourcing/Acquisition) could expand use ofa 
common organizational framework: 


- allows coverage of more of the enterprise or 
potential partnering organizations 


- adapts model features to fit non-developmental 
efforts (e.g., CMMI Services, CMMI Acquisition) 
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3 Complementary “Constellations” 
CMMI-Services 


provides guidance for 
those providing services 
within organizations and 

to external customers 


CMMI-ACQ 
«—/ provides guidance 
to enable 
informed and 
decisive 


acquisition 
leadership 
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Architecture & Constellations 


Development Amplifications 
Development Additions 
ePA XX 


MMI Framework 


eAcquisition Amplifications 
eAcquisition Addition 

*PA YY 

ePA XX 

«PA ACQ 


See BUBB Bee eee | teste ete laslnatestaeateaalelsleateatsleateanaleeialealetaleaalnaletalateatnbeleelnalnbalaateealetenantaleaataletalabetatenabalatslstatsanebalee sean telatalatatatetatet tell tttetet tet alal | 
/ 1. 7 


eServices Amplifications 
eServices Additions 

*PA ZZ 

“PA YY 

*PA SRV 


© 2006 by Carnegie Mellon University 


=— Carnegie Mellon 
—— Software Engineering Institute 


Beyond V1.2, 


First two new “constellations,” CMMI for Services and 
CMMI for Acquisition, have been “commissioned” by 
CMMI Steering Group. Development will be in parallel with 
V1.2 effort; publication sequenced after V1.2 rollout. 


Northrop-Grumman Is leading industry group for CMMI 
Services. 


e Initial focus will be for organizations providing “DoD 
services” as well as internal IT: 


- System maintenance 
- Network Management, IT Services 
- IV&V 
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Beyond V1.2, 


SEI is coordinating development of CMMI-ACQ. 
¢ Will build upon General Motors IT Sourcing expansion 


e Will add government perspectives from both DoD and 
civil agencies 
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Initial CMMI-ACQ Key Acquisition Processes* 


Solicitation & 
Supplier 

Agreement 

Development 


Acquisition 
Management 


Acquisition 
Technical 
Solution 


Acquisition 
Validation 


Acquisition 
Requirements 
Development 


Acquisition 
Verification 


*based on initial CMMI-ACQ model developed by General Motors/SE| 
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Planned Sequence of Models 


CMMI V1.1 ———— 


CMMI-DEV V1.2 
GM IT \ 
Sourcing CMMI-ACQ 


CMMI-AM Laer 
SA-CMM 
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CMMI V1.2...and Beyond 
..the details 
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Topics 

Overview of Model Changes 
Detailed Changes 

Glossary Changes 


Process Area Changes 
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Model Changes 
The model was changed mainly due to the following 

e Reduce complexity and size 

e Expand model coverage 
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Reduce Complexity and Size 


—_—— 
_—— 
—_—— — 
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To reduce model complexity and size, the following 
changes were made: 


e eliminated advanced practices and common features 
¢ eliminated the Supplier Sourcing (SS) addition 


e incorporated Integrated Supplier Management (ISM) 
into Supplier Agreement Management (SAM) 


¢ consolidated and simplified the IPPD material 


e added, modified, and consolidated definitions in the 
glossary (e.g., bidirectional traceability, subprocess) 


¢ adopted a single book approach (i.e., both 
representations are published in one document) 


——-+ — 
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Expand Model Coverage 


To expand model coverage, the following changes were 
made: 


e added hardware amplifications 


e added two work environment specific practices —one in 
Organizational Process Definition (OPD) and one in 
Integrated Project Management (IPM) 


e updated notes and examples to address service 
development and acquisition 


¢ updated the model name to CMMI for Development 
(CMMI-DEV) to reflect the new CMMI architecture 
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Other Significant Model Changes 


Other significant model changes made include 
improved the clarity of the Overview section (Part One) 


added information about and clarified how generic 
practices (GPs) are used 


moved the generic goals and practices to Part Two 


explained how process areas support the 
implementation of the GPs 


added GP elaborations for GP 3.2 


restricted the process areas that can be considered “not 
applicable” to SAM 


added emphasis on project startup in OPF and IPM 
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Topics 

Overview of Model Changes 
Detailed Changes 

Glossary Changes 


Process Area Changes 
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Advanced Practices Eliminated 


There were two types of advanced practices in v1.1: 
¢ those paired with a base practice 
e those that stood alone 
To eliminate advanced practices, the following strategies 
were used: 
e Where a base and advance practice covered the same 
topic, the practices were combined. 
e Where there was only an advanced practice, the 
advanced practice was retained as a specific practice. 
e Specific practice numbering was simplified to exclude 
the capability level. 
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Base and Advanced Practices 
Combined 


The following base and advanced practices were combined 
to form specific practices in v1.2. 


Requirements Development 
e SP 1.1: Elicit Needs (combined with “Collect Stakeholder 


Needs”) 
¢ SP 3.5: Validate Requirements (combined with “Validate 
Requirements with Comprehensive Methods”) 


Technical Solution 
¢ SP 1.1: Develop Alternative Solutions and Selection 
Criteria (combined with “Develop Detailed Alternative 
Solutions and Selection Criteria”) 


e SP 2.3: Design Interfaces Using Criteria (Combined with 
“Establish Interface Descriptions”) 
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Advanced Practices Converted? 


The following advanced practices were retained as 
specific practices in v1.2. 


Requirements Management 

e SP 1.2: Obtain Commitment to Requirements 

¢ SP 1.4: Maintain Bidirectional Traceability of 
Requirements 


Requirements Development 
e SP 3.4: Analyze Requirements to Achieve Balance 


Technical Solution 
e SP 1.2: Select Product Component Solutions 
e SP 2.2: Establish a Technical Data Package 


e SP 2.4: Perform Make, Buy, or Reuse Analyses 
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Advanced Practices Converted 


Product Integration 
e SP 1.2: Establish the Product Integration Environment 


e SP 1.3: Establish Product Integration Procedures and 
Criteria 


Verification 
e SP 1.2: Establish the Verification Environment 


e SP 1.3: Establish Verification Procedures and Criteria 
e SP 2.3: Analyze Peer Review Data 
e SP 3.2: Analyze Verification Results 


Validation 
e SP 1.2: Establish the Validation Environment 


e SP 1.3: Establish Validation Procedures and Criteria 


=— Carnegie Mellon 
—— Software Engineering Institute 


—_— 


No More Common Features 


Discussion of common features was removed from Part 1. 
GPs are no longer organized by common features. 
The common feature headings were removed. 


The generic practices in v1.2 are presented by generic 
goal and are sequentially numbered. 
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Model Structure? 


Continuous V1.1 Continuous V1.2 


Note: no base 
or advanced 
specific 
practices 
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Model Structure? 


Staged V1.1 Staged V1.2 


Common Features 


Note: no 
common 
features 


Carnegie Mellon 
Software Engineering Institute 


No More Supplier Sourcing Addition 


= 
_— 
_—_— ——. 
—_—_—  —--— 
——-> > 
= = 
—_—— 


Supplier Sourcing was eliminated as an addition. 
ISM has been eliminated. 


SAM has been enhanced to contain the unique material 
from ISM. 


Two specific practices were added to Goal 2 in SAM: 
e SP 2.2 - Monitor Selected Supplier Processes 
¢ SP 2.3 - Evaluate Selected Supplier Work Products 
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Simplified IPPD Material 


There are no longer whole process areas that address 
IPPD: 


e removed OEI and moved material to OPD 
e removed IT and moved material to IPM 


Information that addressed “Enable IPPD Management” 
was moved to OPD. 


Information that addressed “Apply IPPD Principles” was 
moved to IPM. 


All IPPD material was condensed and revised to be more 
consistent with the other model material. 
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IPPD Changes Illustrated 
V1.1 V1.2 


SG2 = Enable 
IPPD 
Management 


Process 
Mgt 
PAs 


SG3 = Apply 
IPPD principles 
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The Model Is a Single Document 


All representations, additions, and disciplines are in one 
document. 
Users can choose to use: 


¢ representation-specific content (i.e., continuous, 
staged) 


¢ addition-specific content (i.e., IPPD) 


¢ amplifications (i.e., hardware engineering, software 
engineering, systems engineering) 
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Added Hardware Amplifications and 
Examples 


Six hardware amplifications were created to add emphasis 
on hardware engineering. Here is an example from TS. 


SP 2.1 Design the Product or Product Component 
Develop a design for the product or product component. 


For Hardware Engineering 

Detailed design is focused on product development of 
electronic, mechanical, electro-optical, and other hardware 
products and their components. Electrical schematics and 
interconnection diagrams are developed, mechanical and 
optical assembly models are generated, and fabrication 
and assembly processes are developed. 


Hardware examples were also added to emphasize 
hardware engineering. 
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Added Work Environment Coverage 


Work environment standards are established at the 
organizational level in OPD. 


SP 1.6 Establish Work Environment Standards 
Establish and maintain work environment standards. 
The project’s work environment is established at the 
project level in IPM. 


SP 1.3 Establish the Project's Work Environment 
Establish and maintain the project’s work environment 
based on the organization’s work environment standards. 
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Other Specific Practice Changes 


OID, SP 1.4 

Select process and technology improvements [not 
“improvement proposals”] for deployment across the 
organization. 


OPP, oP lal 

Select the processes or subprocesses [not “process 
elements’] in the organization’s set of standard processes 
that are to be included in the organization’s process- 
performance analysis. 
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Overview Section Improvements 


The following improvements were made to the model 
overview (i.e., Part One): 


¢ The chapter containing the generic goals and practices 
was moved to Part Two with the process areas. 

e All definitions are consolidated into the glossary. 

e Chapters were reordered into a more logical sequence. 

e The Preface and Using CMMI Models chapter were 
rewritten and updated. 


¢ Descriptions were updated to reflect the new CMMI 
architecture: 
- Added descriptions of constellations and additions 


- Removed descriptions of base and advanced 
practices and common features 
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Improved Generic Practices* 


Editorial changes were made to the generic practices. 
These slides highlight the changes that affect the content. 


GP 1.1: Perform Specific Practices 
The practice title and statement changed from “perform 
base practices” to “perform specific practices.” 


GP 2.2: Plan the Process 
The informative material was condensed to be consistent 
with the other generic practices. 


GP 2.4: Assign Responsibility 
In the informative material “and authority” was added. 
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Improved Generic Practices 


GP 2.6: Manage Configurations 
In the GP statement, “levels of configuration management 


was changed to “levels of control 
GP 2.9 Objectively Evaluate Adherence 
Added informative material to emphasize work products 
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also. 
GP 5.2: Correct Root Causes of Problems 
Added notes that the focus of this GP is on a quantitatively 
managed process, though root causes may be found 


outside of that process 


2006 by Carnegie Mellon University 


Carnegie Mellon 
Software Engineering Institute 


Explained Generic Practices Better 
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Moved generic goals and practices to Part Two with the 
process areas so that all normative elements of the model 
are consolidated in one place 


Added information about how process areas support the 
implementation of generic practices (GPs) 


Added GP elaborations for GP 3.2 
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“Not Applicable” Process Areas 


The set of PAs evaluated to achieve a maturity level is an 
important variable when conducting an appraisal. In v1.1 it 
was not clear which PAs could be considered “not 
applicable.” 


In v1.2, the guidance for appraisals exists in both 
SCAMPIS™ MDD Appendix A and SCAMPI A Appraisal 
Disclosure Statement (ADS): 


¢ Only SAM can be declared not applicable. 


¢ Decisions on PAs included in the appraisal must be 
made by the lead appraiser in conjunction with the 
appraisal sponsor. 


e Rationale for declaring SAM to be “not applicable” must 
be provided in the Appraisal Disclosure Statement. 
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Topics 

Overview of Model Changes 
Detailed Changes 

Glossary Changes 


Process Area Changes 
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Glossary Changes 


The following slides contain significant changes to glossary 
definitions. Definitions that only had editorial changes are not 
included. 


New definitions: addition, amplification, bidirectional traceability, 
customer requirement, data, functional configuration audit, 
hardware engineering, higher level management, physical 
configuration audit, project startup, and service. 


Revised definitions: acquisition, appraisal, appraisal findings, 
appraisal scope, audit, capability evaluation, configuration audit, 
customer, data management, establish and maintain, generic 
goal, objective evidence, process element, product, product 
component, project, quality- and process-performance 
objectives, requirements traceability, shared vision, subprocess, 
traceability, and work product. 


=— Carnegie Mellon 
—— Software Engineering Institute 


_— 


Definitions Deleted From the Glossary 


Deleted definitions: ability to perform, advanced practices, 
agreement/contract requirements, appraisal tailoring, 
appraisal team leader, base practices, CMMI model 
tailoring, commitment to perform, directing 
implementation, discipline amplification, lead appraiser, 
process context, solicitation package, strength, verifying 
implementation, weakness 


Many of these definitions were deleted because the term 
wasn’t used in the model or the overall concept was 
removed. 
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New Definitions? 


addition 

In the CMMI Product Suite, a clearly marked model 
component that contains information of interest to 
particular users. In a CMMI model, all additions bearing 
the same name (e.g., the IPPD addition) may be optionally 
selected as a group for use. 


amplification 

Amplifications are informative model components that 
contain information relevant to a particular discipline. For 
example, to find an amplification for software engineering, 
you would look in the model for items labeled “For 
Software Engineering.” The same is true for other 
disciplines. 
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New Definitions? 


bidirectional traceability 

An association among two or more logical entities that is 
discernable in either direction (i.e., to and from an entity). 
(See also "requirements traceability" and "traceability.") 


customer requirement 

The result of eliciting, consolidating, and resolving conflicts 
among the needs, expectations, constraints, and 
interfaces of the product's relevant stakeholders in a way 
that is acceptable to the customer. (See also “customer.”) 
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New Definitions? 


data 

Recorded information, regardless of the form or method of 
recording, including technical data, computer software 
documents, financial information, management information, 
representation of facts, numbers, or datum of any nature that 
can be communicated, stored, and processed. 


functional configuration audit 

An audit conducted to verify that the development of a 
configuration item has been completed satisfactorily, that the 
item has achieved the performance and functional 
characteristics specified in the functional or allocated 
configuration identification, and that its operational and 
Support documents are complete and satisfactory. (See also 
"configuration audit," "configuration management,” and 
“physical configuration audit.") 
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New Definitions’ 


hardware engineering 

The application of a systematic, disciplined, and 
quantifiable approach to transform a set of requirements 
representing the collection of stakeholder needs, 
expectations, and constraints using documented 
techniques and technology to design, implement, and 
maintain a tangible product. (See also "software 
engineering" and "systems engineering.") 


In CMMI, hardware engineering represents all technical 
fields (e.g., electrical or mechanical) that transform 
requirements and ideas into tangible and producible 
products . 
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New Definitions? 


higher level management 

The person or persons who provide the policy and overall 
guidance for the process, but do not provide the direct 
day-to-day monitoring and controlling of the process. Such 
persons belong to a level of management in the 
organization above the immediate level responsible for the 
process and can be (but are not necessarily) senior 
managers. (See also "senior manager.") 


physical configuration audit 

An audit conducted to verify that a configuration item, as 
built, conforms to the technical documentation that defines 
and describes it. (See also, "configuration audit," 
"configuration management," and "functional configuration 
audit.") 
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New Definitions® 


project startup 

When a set of interrelated resources are directed to 
develop or deliver one or more products for a customer or 
end user. (See also “project.”) 


service 

In the CMMI Product Suite, a service is a product that is 
intangible and non-storable. (See also "product," 
“customer,” and "work product.") 
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Revised Definitions* 


acquisition 
The process of obtaining products (goods and services) 
through contract. 


appraisal 

In the CMMI Product Suite, an examination of one or more 
processes by a trained team of professionals using an 
appraisal reference model as the basis for determining, at 
a minimum, strengths and weaknesses. (See also 
“assessment” and “capability evaluation.”) 


appraisal findings 

The results of an appraisal that identify the most important 
issues, problems, or opportunities for process 
improvement within the appraisal scope. Appraisal 
findings are inferences drawn from corroborated objective 
evidence. 
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Revised Definitions 


appraisal scope 

The definition of the boundaries of the appraisal 
encompassing the organizational limits and the CMMI 
model limits within which the processes to be investigated 
operate. 


audit 

In CMMI process improvement work, an objective 
examination of a work product or set of work products 
against specific criteria (e.g., requirements). 


— Carnegie Mellon 
—— Software Engineering Institute 


Revised Definitions? 


capability evaluation 

An appraisal by a trained team of professionals used as a 
discriminator to select suppliers, to monitor suppliers against 
the contract, or to determine and enforce incentives. 
Evaluations are used to gain insight into the process 
capability of a supplier organization and are intended to help 
decision makers make better acquisition decisions, improve 
subcontractor performance, and provide insight to a 
purchasing organization. (See also “appraisal” and 
“assessment.”) 


configuration audit 

An audit conducted to verify that a configuration item, or a 
collection of configuration items that make up a baseline, 
conforms to a specified standard or requirement. (See also 
“audit,” “configuration item,” "functional configuration audit," 
and "physical configuration audit.") 
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Revised Definitions’ 


customer 

The party (individual, project, or organization) responsible 
for accepting the product or for authorizing payment. The 
customer is external to the project (except possibly when 
integrated teams are used, as in IPPD), but not 
necessarily external to the organization. The customer 
may be a higher level project. Customers are a subset of 
stakeholders. (See also “stakeholder.”) 


In most cases where this term is used, the preceding 
definition is intended; however, in some contexts, the term 
"customer" is intended to include other relevant 
stakeholders. (See also “customer requirement.”) 
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~ Revised Definitions® 


data management 

The disciplined processes and systems that plan for, 
acquire, and provide stewardship for business and 
technical data, consistent with data requirements, 
throughout the data lifecycle. 


establish and maintain 

In the CMMI Product Suite, you will encounter goals and 
practices that include the phrase “establish and maintain. 
This phrase means more than a combination of its 
component terms; it includes documentation and usage. 
For example, “Establish and maintain an organizational 
policy for planning and performing the organizational 
process focus process” means that not only must a policy 
be formulated, but it also must be documented, and it 
must be used throughout the organization. 
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Revised Definitions® 


generic goal 

A required model component that describes the 
characteristics that must be present to institutionalize the 
processes that implement a process area. (See also 
“institutionalization.”) 


objective evidence 

As used in CMMI appraisal materials, documents or 
interview results used as indicators of the implementation 
or institutionalization of model practices. Sources of 
objective evidence can include instruments, presentations, 
documents, and interviews. 
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Revised Definitions’ 


process element 

The fundamental unit of a process. A process can be 
defined in terms of subprocesses or process elements. A 
Subprocess can be further decomposed into subprocesses 
or process elements; a process element cannot. (See also 
"process" and "subprocess.") 


Each process element covers a closely related set of 
activities (e.g., estimating element and peer review 
element). Process elements can be portrayed using 
templates to be completed, abstractions to be refined, or 
descriptions to be modified or used. A process element 
can be an activity or task. 
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Revised Definitions® 


product 

In the CMMI Product Suite, a work product that is intended 
for delivery to a customer or end user. The form ofa 
product can vary in different contexts. (See also 
“customer,” "product component," "service," and “work 
product.”) 


product component 

In the CMMI Product Suite, a work product that is a lower 
level component of the product. Product components are 
integrated to produce the product. There may be multiple 
levels of product components. (See also “product” and 
“work product.”) 
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Revised Definitions? 


project 

In the CMMI Product Suite, a managed set of interrelated 
resources which delivers one or more products to a 
customer or end user. A project has a definite beginning 
(l.e€., project startup) and typically operates according to a 
plan. Such a plan is frequently documented and specifies 
what is to be delivered or implemented, the resources and 
funds to be used, the work to be done, and a schedule for 
doing the work. A project can be composed of projects. 
(See also “project startup.”) 
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Revised Definitions?® 


quality and process-performance objectives 

Objectives and requirements for product quality, service 
quality, and process performance. Process-performance 
objectives include quality; however, to emphasize the 
importance of quality in the CMMI Product Suite, the 
phrase quality and process-performance objectives is 
used rather than just process-performance objectives. 


requirements traceability 

A discernable association between requirements and 
related requirements, implementations, and verifications. 
(See also "bidirectional traceability" and "traceability.") 
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Revised Definitions" 


shared vision 

A common understanding of guiding principles including 
mission, objectives, expected behavior, values, and final 
outcomes, which are developed and used by a project. 


Subprocess 

A process that is part of a larger process. A subprocess 
can be decomposed into subprocesses and/or process 
elements. (See also “process," "process description,” and 
"process element.") 


traceability 

A discernable association among two or more logical 
entities such as requirements, system elements, 
verifications, or tasks. (See also "bidirectional traceability 
and “requirements traceability.”) 
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Revised Definitions? 


work product 

In the CMMI Product Suite, a useful result of a process. 
This can include files, documents, products, parts of a 
product, services, process descriptions, specifications, 
and invoices. A key distinction between a work product 
and a product component is that a work product is not 
necessarily part of the product. (See also “product” and 
“oroduct component.”) 


In CMMI models, you will see the phrase work products 
and services. Even though the definition of work product 
includes services, this phrase is used to emphasize the 
inclusion of services in the discussion. 
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Topics 

Overview of Model Changes 
Detailed Changes 

Glossary Changes 


Process Area Changes 
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Process Area Improvements? 


Improvements were made to all process areas; some 
process areas changed more than others. Only the 
process areas that were changed significantly will be 
addressed. 


Many of these changes were discussed earlier. However, 
these slides show you significant changes by process 
area. 
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Process Area Improvements? 


The following process areas were improved significantly: 
e Integrated Project Management +IPPD (IPM+IPPD) 
¢ Organizational Process Definition +IPPD (OPD+IPPD) 
¢ Organizational Process Focus (OPF) 
e Requirements Management (REQM) 
e Requirements Development (RD) 
e Supplier Agreement Management (SAM) 
¢ Technical Solution (TS) 
¢ Validation (VAL) 
¢ Verification (VER) 
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ntegrated Project Management +IPPD! 


Specific Goal Specific Practice 
Use the Project’s 1.1 — Establish the Project’s Defined 
Defined Process Process 


1.2 - Use Organizational Process Assets 
for Planning Project Activities 


1.3 — Establish the Project’s Work 
Environment 


1.4 - Integrate Plans 


1.5 —- Manage the Project Using the 
Integrated Plans 


1.6 — Contribute to the Organizational 
Process Assets 


¢ Modified SP 1.1 from “Establish and maintain the project’s defined 
process” to “Establish and maintain the project’s defined process from 
project startup through the life of the project.” 


¢ Added SP 1.3 “Establish the Project’s Work Environment.” (This 
practice is new to CMMI.) 
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ntegrated Project Management +IPPD2 


Specific Goal Specific Practice 
Coordinate and 2.1 —- Manage Stakeholder Involvement 
Collaborate with 2.2 - Manage Dependencies 
Relevant Stakeholders 2.3 - Resolve Coordination Issues 
Apply IPPD Principles 3.1 - Establish the Project’s Shared 
Vision 
3.2 — Establish the Integrated Team 
Structure 


3.3 — Allocate Requirements to 
Integrated Teams 


3.4 - Establish Integrated Teams 


3.5 - Ensure Collaboration among 
Interfacing Teams 
¢ Reduced the IPPD Addition to one goal (SG3 “Apply IPPD Principles”) 
and its practices. 
¢ TO emphasize the IPPD Addition, the name of this process area is now 
“Integrated Project Management +IPPD” or “IPM +IPPD.” 
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Organizational Process Definition 
+IPPD! 


Specific Goal Specific Practice 
Establish Organizational 1.1 - Establish Standard Processes 
Process Assets 1.2 - Establish Lifecycle Model 


Descriptions 


1.3 - Establish Tailoring Criteria and 
Guidelines 


1.4 - Establish the Organization’s 
Measurement Repository 


1.5 — Establish the Organization’s Process 
Asset Library 


1.6 — Establish Work Environment 
Standards 


¢ Added “and work environment standards” to the purpose statement. 


e Added SP 1.6 “Establish Work Environment Standards.” (This practice 
is new to CMMI.) 
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Organizational Process Definition 
+IPPD2 


Specific Goal Specific Practice 


Enable IPPD Management 2.1 - Establish Empowerment 
Mechanisms 


2.2 — Establish Rules and Guidelines for 
Integrated Teams 


2.3 — Balance Team and Home 
Organization Responsibilities 


e Added an IPPD Addition to OPD (SG2 “Enable IPPD Management” and 
its practices). 


¢ To emphasize the IPPD Addition, the name the process area is now 
“Organizational Process Definition +IPPD” or “OPD +IPPD.” 
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Organizational Process Focus?* 


Specific Goal Specific Practice 
Determine Process 1.1 - Establish Organizational Process 
Improvement Needs 


Opportunities 1.2 — Appraise the Organization’s 


Processes 


1.3 — Identify the Organization’s 
Process Improvements 


¢ Modified the purpose statement to emphasize deployment. 


¢ SP 1.2 “Appraise the organization’s processes periodically and as 
needed to maintain an understanding of their strengths and 
weaknesses.” uses “organization’s processes” instead of “processes of 
the organization.” 
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Organizational Process Focus? 
Specific Goal Specific Practice 
2.1 — Establish Process Action Plans 
Implement Process Action Plans 


Plan and Implement 
2.2 - 


Process 
Improvements 


¢ Modified SG2 from “Plan and Implement Process Improvement 
Activities” to “Plan and Implement Process Improvements 
¢ Moved to a new SG3 and modified what were SP 2.3 and SP 2.4 in 
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Organizational Process Focus? 


Specific Goal Specific Practice 

Deploy Organizational 3.1 - Deploy Organizational Process 
Process Assets and Assets 

Incorporate Lessons 3.2 - Deploy Standard Processes 
Learned 3.3 - Monitor Implementation 


3.4 — Incorporate Process-Related 
Experiences into the 
Organizational Process Assets 


e Added new SG3, “Deploy Organizational Process Assets and 
Incorporate Lessons Learned.” 


e Moved what were SP 2.3 and SP 2.4 in v1.1 to the new SG3 as SP 3.1 
and SP 3.4. 


¢ Added two new SPs: SP 3.2 “Deploy Standard Processes,” and SP 3.3 
“Monitor Implementation.” 
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Requirements Management 


Specific Goal Specific Practice 


Manage Requirements 1.1 - Obtain an Understanding of 
Requirements 


1.2 — Obtain Commitment to 
Requirements 


1.3 - Manage Requirements Changes 


1.4 —- Maintain Bidirectional 
Traceability of Requirements 


1.5 — Identify Inconsistencies Between 
Project Work and Requirements 


¢V1.2 REQM SP 1.4 practice statement now reads, “Maintain 
bidirectional traceability among the requirements and work products. 


e Project plans are no longer mentioned in this SP statement. 


¢ The description of bidirectional traceability is improved as is Its 
definition in the glossary. 
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Requirements Development? 


Specific Goal Specific Practice 
Develop Customer 1.1 - Elicit Needs 


Requirements 1.2 — Develop the Customer 
Requirements 


Develop Product 2.1 - Establish Product and Product 
Requirements Component Requirements 


2.2 — Allocate Product Component 
Requirements 


2.3 — Identify Interface Requirements 


¢ Former base practice “Collect Stakeholder Needs” is eliminated and 
former advanced practice, “Elicit Needs” is kept. 


¢ Informative text is added to the introductory notes about applying RD 
to maintenance projects. 
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Requirements Development? 


Specific Goal Specific Practice 
Analyze and Validate 3.1 - Establish Operational Concepts 
Requirements and Scenarios 


3.2 — Establish a Definition of Required 
Functionality 


3.3 — Analyze Requirements 


3.4 —- Analyze Requirements to Achieve 
Balance 


3.5 — Validate Requirements 


¢ Material from V1.1 TS SP 1.2, “Evolve Operational Concepts and 
Scenarios,” is incorporated into RD SP 3.1. 


e Material from V1.1 RD SP 3.5-1, “Validate Requirements,” and RD 
SP 3.5-2, “Validate Requirements with Comprehensive Methods” 
were consolidated into a single practice. 
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Supplier Agreement Management 
Specific Goal Specific Practice 
Establish Supplier 1.1 - Determine Acquisition Type 
Agreements 1.2 - Select Suppliers 
1.3 - Establish Supplier Agreements 
Satisfy Supplier 2.1 — Execute the Supplier Agreement 
Agreements 2.2 — Monitor Selected Supplier 
Processes 
2.3 — Evaluate Selected Supplier Work 
Products 


2.4 — Accept the Acquired Product 
2.5 — Transition Products 
¢V1.1 SAM SP2.1 “Review COTS Products,” was eliminated. “Identify 
candidate COTS products that satisfy requirements” is a new 


subpractice under the Technical Solutions Process Area SP1.1, 
“Develop Alternative Solutions and Selection Criteria.” 


¢ SP2.2 and SP2.3 were added because ISM was eliminated. 
¢ The purpose of SAM was also updated. 
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Technical Solution? 
Specific Goal Specific Practice 
Select Product- 1.1 - Develop Alternative Solutions and 
Component Solutions Selection Criteria 
1.2 - Select Product-Component 
Solutions 


¢V1.1 TS SP 1.1-1, “Develop Alternative Solutions and Selection 
Criteria,” and TS SP 1.1-2, “Develop Detailed Alternative Solutions and 


Selection Criteria” are consolidated into a single practice. 
¢ “Identify candidate COTS products that satisfy requirements” is a new 
subpractice under SP1.1. 


¢V/1.1 TS SP 1.2 “Evolve Operational Concepts and Scenarios” is 
incorporated into RD SP 3.1, “Establish Operational Concepts and 


Scenarios.” 
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Technical Solution2 


Specific Goal Specific Practice 
Develop the Design 2.1 — Design the Product or Product 
Component 
2.2 — Establish a Technical Data 
Package 


2.3 — Design Interfaces Using Criteria 
2.4 —- Perform Make, Buy, or Reuse 


Analyses 
Implement the 3.1 — Implement the Design 
Product Design 3.2 — Develop Product Support 


Documentation 


¢V1.1 TS SP 2.3-1, “Establish Interface Descriptions,” and TS SP 2.3- 
3, “Design Interfaces Using Criteria” are consolidated into a single 
practice. 
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Validation 
Specific Goal Specific Practice 
Prepare for Validation 1.1 - Select Products for Validation 
1.2 — Establish the Validation 
Environment 
1.3 — Establish Validation Procedures 
and Criteria 
Validate Product or 2.1 - Perform Validation 
Product Components 2.2 - Analyze Validation Results 


e Notes were added to VAL to stress that validation activities are 
performed incrementally and involve relevant stakeholders. 


¢ The phrase “and identify issues” was deleted from the statement of SP 
2.2 “Analyze Validation Results” to maintain parallelism with VER SP 
3.2 “Analyze Verification Results.” 
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Verification! 
Specific Goal Specific Practice 
Prepare for Verification 1.1 — Select Work Products for 
Verification 
1.2 — Establish the Verification 
Environment 
1.3 — Establish Verification Procedures 
and Criteria 
Perform Peer Reviews 2.1 —- Prepare for Peer Reviews 


2.2 —- Conduct Peer Reviews 
2.3 — Analyze Peer Review Data 


¢ No changes to SG1, SG2, or their practices. 
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Verification? 
Specific Practice 


Specific Goal 
3.1 - Perform Verification 
3.2 — Analyze Verification Results 


Verify Selected Work 


Products 
¢ The phrase “and identify corrective action” was deleted from both 


the title and statement of SP 3.2 “Analyze Verification Results. 
(Corrective action is handled in PMC SG2, “Manage Corrective 


Action to Closure.) 
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Summary 


Many changes were made to the CMMI models to improve 
quality. The major changes include 


¢ name changed to “CMMI for Development” 

¢ both representations in one document 

¢ amplifications improved; added hardware amplifications 
* common features and advanced practices eliminated 

¢ SS addition eliminated; ISM brought into SAM 

¢ guidelines for “not applicable” process areas clarified 

° overview and glossary improved 

¢ work environment material added to OPD and IPM 

e IPPD material simplified and consolidated 

° process deployment strengthened in IPM and OPF 
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Applying CMMI in Small Settings 


Where are we with our work in small settings? 
¢ completed technical feasibility pilots in Huntsville, 
Alabama with two small companies in the US Army supply 
chain 


¢ posted the toolkit from this pilot for review: 


- http:/www.sei.cmu.edu/ttp/publications/toolkit 


¢ chartered a project to further research in and evolve 
guidance for CMMI in Small Settings (CSS) 


Where are we going? 
¢ International Research Workshop for Process 
Improvement in Small Settings held October 19-20, 2005 


° call for Interest in CSS project is posted on SEI web: 


- http://www.sei.cmu.edu/cmmi/acss/participation.html 
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The purpose of this module is to describe the major 
changes to the SCAMPI A appraisal method for v1.2. 
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Revision Process 


The CMMI Steering Group provided a long-term strategy 
and the upgrade criteria for v1.2. 


The SCAMPI Upgrade Team (SUT) reviewed change 
requests to identify possible changes for the v1.2 
appraisal method documents: Appraisal Requirements for 
CMMI (ARC) and Method Definition Document (MDD). 


The CMMI Steering Group served as the configuration 
control board for v1.2 changes to the ARC and MDD. 


The SUT developed a draft of the ARC and MDD for 
review by lead appraisers in May 2006. 


The ARC and MDD were released as part of the v1.2 
CMMI Product Suite. 
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SCAMPI A V1.2 Major Themes 


Reduce complexity and ambiguity 

Provide additional guidance where needed 
Strengthen appraisal planning and conduct 
Strengthen appraisal reporting 

Define appraisal validity period 


Strengthen lead appraiser requirements 
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Reduce Complexity* 


The requirement for instruments (e.g., questionnaires) was 
removed. 


Only two types of objective evidence are now required: 
¢ documents 
e interviews 


The following sections in MDD were revised: 

e switched 2.2 Verify and Validate Objective Evidence 
and 2.3 Document Objective Evidence so that the order 
of tasks reflects the natural order of conducting an 
appraisal 

e separated Verify Objective Evidence and Validate 
Preliminary Findings to better describe each process 
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Reduce Complexity 


The use of the term instantiation was changed: 


e Instantiation is now defined as “the implementation of a 


model practice used in the appropriate context within 
the boundaries of an organizational unit.” 


¢ The word “instantiation” for project and organizational- 
wide entities was replaced with “project” or “support 
group.” 
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Reduce Ambiguity 


The rating Not Rated was clarified: 


e Process areas outside of the model scope are rated as Out of 
Scope. For example, for a maturity level 3 appraisal, maturity 
level 4 and 5 process areas are rated as Out of Scope. 

e For process areas that have insufficient data to be rated, the 
rating is Not Rated. 

¢ Process areas in the model scope, but outside the 
organizational scope are rated as Not Applicable. The only 
process area that can be Not Applicable is SAM (as 
determined by the appraisal team). 


The practice characterization tables were revised: 
¢ clarified the use of virtual versus live interviews 
¢ changed “face-to-face interviews” to “oral interviews” 
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Provide Additional Guidance 


Guidance for readiness reviews was revised to include 
team and logistical readiness. 


Additional guidance was provided for using virtual 
methods (e.g., for interviews and briefings). 


Additional guidance was provided for using alternative 
practices (i.e., Appendix C: Alternative Practice 
Identification and Characterization Guidance). 
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Strengthen Appraisal Planning 
and Conduct 


Organizational unit sampling was revised to* 

e strengthen parameters and limits for organizational 
Sampling (e.g., identifying a minimum number of focus 
projects) 

e include additional criteria for reporting sampling 
decisions 


The Conduct Appraisal Phase must now be complete 
within 90 days. 


Appraisal team members are now required to sign final 
findings. 


* Changes to address sampling were extensive. Refer to the MDD for details. 
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Strengthen Appraisal Reporting 


The Appraisal Disclosure Statement (ADS) now requires the 
following information. 


Organizational sampling criteria and decisions (e.g., projects 
included, projects excluded, percentage of organization 
represented) 


Basis for maturity/capability level 4 and 5 appraisal results 
¢ subprocesses statistically managed 
¢ mapping of these subprocesses to quality and process- 
performance objectives 


Signatures of both the lead appraiser and sponsor 
e The lead appraiser affirms that the appraisal scope is 
representative of the organizational unit. 
e The sponsor affirms the accuracy of the ADS and authorizes 
the SEI to conduct any audits deemed necessary. 
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Define Appraisal Validity Period 
V1.2 appraisal results are valid for a maximum of 3 years 
from the date of the ADS 
V1.1 appraisals will expire on August 31, 2007 or 3 years 
after the date the appraisal was conducted, whichever is 


later. 
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Strengthen Lead Appraiser 
Requirements 


Prior to conducting a v1.2 SCAMPI appraisal, the following 
must Occur: 

e Current candidate and authorized lead appraisers and 
team leaders must complete CMMI v1.2 Upgrade 
Training. 

e Candidate and authorized lead appraisers must attend 
SCAMPI Face-to-Face Training. 

e Those who want to conduct v1.2 SCAMPI level 4 or 5 
appraisals must be certified. Certification requirements 
address the following: 

- education, training, and experience in level 4 and 5 
concepts 
- completion of an oral exam 
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Summary 


The SCAMPI A appraisal method was revised based on 
change requests received to 

¢ reduce complexity and ambiguity 

¢ provide additional guidance where needed 

e strengthen appraisal planning and conduct 

e strengthen appraisal reporting 

¢ define the appraisal validity period 

e strengthen lead appraiser requirements 


The changes are intended to make appraisals more 
accurate, reliable, and efficient. 
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The purpose of this module is to describe the major 
changes to the CMMI Training for v1.2. 
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SEI Training for CMMI 


Introduction 
to CMMI 
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Upgrade Training 
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CMMI Training Changes 


The Introduction to CMMI course was updated to address 
change requests and to address CMMI Product Suite v1.2 
changes. 


The following courses will be updated in Fall 2006 to 
address change requests and CMMI Product Suite v1.2 
changes: 

e Intermediate Concepts of CMMI 

e CMMI Instructor Training 

¢ SCAMPI Lead Appraiser Training 

¢ SCAMPI B and C Team Leader Training 


CMMI v1.2 Upgrade Training was also developed to help 
users move from v1.1 to v1.2 
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Topics 

Introduction to CMMI 
Intermediate Concepts of CMMI 
Instructor Training 

SCAMPI Training 


Examinations 
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Major Changes to the 
Introduction to CMMI Course 


The major changes to the Introduction to CMMI, v1.2 
course focused on three areas: 

* course content 

° exercises 

* course reference 
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Course Content? 
The benefits associated with adopting CMMI were 


updated. 
Information about CMMI benefits is available in the SEI 
technical report, Performance Results of CMMI-Based 
Process Improvement (CMU/SEI-2006-TR-004). It is 
available on the SEI Web site at 
http://www.sei.cmu.edu/publications/documents/ 
06.reports/O6trO04.html. 
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Course Content? 


Material related to the following was removed: 
e advanced and base practices 
¢ common features 


Material related to the following was added or changed: 

¢ How PAs support the implementation of generic 
practices was updated. (The basis for this material can 
be found in Figure 6.2 in the model.) 

¢e The “Sampling Generic Practices” slides now 
collectively include all generic practices, and reference 
capability and maturity levels. 

e Information on the CMMI architecture was added. 
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Course Content? 


These changes were also made to reflect changes in the model: 


Integrated Supplier Management was removed and two 
specific practices were moved to Supplier Agreement 
Management. 

Organizational Environment for Integration and Integrated 
Teaming were removed and material was moved to 
Organizational Process Definition and Integrated Project 
Management. 

Two new practices were added to cover work environment— 
one in Organizational Process Definition and one in Integrated 
Project Management. 

A third goal was added to Organizational Process Focus to 
address the deployment of organizational process assets. 


There are now 22 PAs discussed in the course. 


Carnegie Mellon 
Software Engineering Institute 


Exercises 


Exercise 1, “Important Process Improvement Ideas,” and 
Exercise 7, “Measurement Implications of Your Process 


ilhh 


Improvement Goals,” were simplified. 
Exercises were updated to reflect the removal of some 


PAs. 
Other minor editorial changes were made. 
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Course Reference 


The course was designed so that the reference model 


e the technical report, CMMI for Development, Version 


used can be either 
2 

e the Addison-Wesley book, CMMI: Guidelines for 

Process Integration and Product Improvement, Second 


Edition 
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Topics 

Introduction to CMMI 
Intermediate Concepts of CMMI 
Instructor Training 

SCAMPI Training 


Examinations 
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Major Changes to Intermediate 
Concepts of CMMI 


The major changes to the Intermediate Concepts of CMMI 
course will focus on four areas: 

¢ course format 

° exercises 

e student presentations 

° course content 


Editorial changes also will be made to correct identified 
errors. 
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Course Format 


The format of the Intermediate Concepts of CMMI course 
will change. 


In the v1.1 course, the instructor presented material the 
first day and on following days the students presented. In 
v1.2, student presentations, exercises, and SEI instructor 
presentations will be intermixed throughout the week to 
provide a better balance. 


TOEFL* requirements will now be used as part of the 
registration process. 


*Test of English as a Foreign Language 
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Course Content 
The majority of changes to the course slides will reflect the 


changes from CMMI v1.2 model 
In addition, the following changes will be made 
e Material on application and interpretation will be added 


e There will be more emphasis on institutionalization (i.e 


generic goals and generic practices) 
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Exercises 


All exercises will be updated to better meet the course 
objectives. 


Several exercises will be integrated with a common theme 
so that concepts are developed using related exercises 
that build on each other throughout the course. 
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Student Presentations 


The student presentations will be modified to better meet 
the course objectives. 

e The process used for assigning student presentation 
topics will be revised. More information will be 
collected from students to help to better assign the PAs. 

e The length of student presentations will be reduced. 

e Slides for instructors will be provided to reinforce the 
key points of each PA. 
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Topics 

Introduction to CMMI 
Intermediate Concepts of CMMI 
Instructor Training 

SCAMPI Training 


Examinations 


=— Carnegie Mellon 
~—— Software Engineering Institute 


Major Changes to Instructor Training 
The major changes to the Instructor Training course will 
focus on two areas: 


* course content 
¢ student presentations 
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Course Content 


Changes to all course materials will be made to reflect the 
changes to: 

e the Introduction to CMMI course 

e the CMMI v1.2 model 


The sections that will be revised include: 
e the course structure and key concepts section 
e the review of CMMI topics section 


Editorial changes will be made to correct identified errors. 
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Student Presentations 


The following changes will be made: 

e The assignments will be updated to reflect changes in 
the model (e.g., delete presentation assignments 
related to the PAs that were removed from the model). 

e Add a teaching assignment focused on topics such as 

- model components 

- generic goals and generic practices 
- capability levels and maturity levels 
- benefits of process improvement 

- PA-to-PA relationships 
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Topics 

Introduction to CMMI 
Intermediate Concepts of CMMI 
Instructor Training 

SCAMPI Training 


Examinations 
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Major Changes to SCAMPI Training 
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The changes to the SCAMPI Lead Appraiser Training and 
SCAMPIB and C Team Leader Training courses will focus 
on two areas: 

° course content 

° exercises 
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Course Content 


The majority of changes to the course slides will reflect the 
changes from SCAMPI v1.2. 


In addition, material will be added on ethics and model 
interpretation. 
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Exercises 


The majority of changes to the course exercises will be 
made to reflect the CMMI for Development, Version 1.2. 
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Topics 

Introduction to CMMI 
Intermediate Concepts of CMMI 
Instructor Training 

SCAMPI Training 


Examinations 
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Examinations 


The construction and format of examinations have 
changed. v1.1 tests were largely short answer tests that 
were the same for all students. 


For v1.2, tests will be generated from an item bank and 
now will be multiple choice. CMMI v1.2 Upgrade Training 
for Instructors, Lead Appraisers, and Team Leaders is the 
first course to use this approach. The Intermediate 
Concepts of CMMI and Instructor Training will follow. 
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Multiple Choice Examinations 


This new approach, using an item bank and multiple 
choice questions, allows multiple versions of examinations 
that can be constructed more easily: 
e The sequence of multiple choice responses can vary 
from test to test. 
e The order of questions can vary from test to test. 
e The selection of questions can vary from test to test, but 
cover the same categories. 


This new approach allows us to 
e add, modify, and delete questions from the test more 
easily 
e better evaluate the student’s knowledge 
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Summary 
Many changes were made to the CMMI training courses to 
improve quality and reflect changes to the CMMI Product 


Suite: 


e CMMI for Development, Version 1.2 
¢ SCAMPI v1.2 
e Introduction to CMMI, v1.2 
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For More Information... 


For more information about CMMI 
¢ http://www.sei.cmu.edu/cmmi/ (main CMMI site) 


Other Web sites of interest include 


¢ http://seir.sei.cmu.edu/seir/ (Software Engineering 
Information Repository) 


¢ http://dtic.mil/ndia (annual CMMI Technology 
Conferences) 


¢ http://seir.sei.cmu.edu/pars (publicly released SCAMPI 
appraisal Summaries) 


e https://oscw.sei.cmu.edu/pub/bscw.cgi/0/79783 


Or, contact 

SEI Customer Relations 

Phone: 412 / 268-5800 

Email: customer-relations@sel.cmu.edu 
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Organization Size 
Based on the total number of employees within the area of the organization 


that was appraised 
1001 to 2000 — 
6.3% \ _ 2000+ 
\ / 3.3% 
501 to ain _ 250r fewer 
9.1% a ———— 9.9% 

26 to 50 
12.7% 


301 to 500 _ 
: 201 to 2000+ 
. : =o 1 to 100 
: a 
a _ 51to75 
10.3% 
101%46-200 
18.9% 


201to300 
76 to 100 
8.7% 


11.0% 


Based 0n1,083 organizations reporting size data 
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Appraisals and Maturity Levels by Country 


Maturity Maturity Maturity Maturity Maturity Maturity Maturity Maturity Maturity Maturity 


Number of Level1 Level2 Level3 Level4  Level5 Number of Level1 Level2 Level3 Level4  Level5 
Appraisals Reported Reported Reported Reported Reported Country Appraisals Reported Reported Reported Reported Reported 


Argentina 12 No Yes Yes Yes No Latvia 10 or fewer 

Australia (21 Yes Yes Yes Yes Yes Malaysia 10 or fewer 

Austria 10 or fewer Mauritius 10 or fewer 

Belarus 10 or fewer Mexico 10 or fewer 

Belgium 10 or fewer Netherlands 10 or fewer 

Brazil (22 No Yes Yes No Yes New Zealand 10 or fewer 

Canada 15 No Yes Yes No Yes Philippines "10 

Chile 10 or fewer Portugal 10 or fewer 

China "17 Yes Yes Yes Yes Yes Russia 10 or fewer 

Colombia 10 or fewer Singapore 10 or fewer 

Czech Republic — 10 or fewer Slovakia 10 or fewer 

Denmark 10 or fewer South Africa 10 or fewer 

Egypt 10 or fewer Spain 18 No Yes Yes No Yes 
Finland 10 or fewer Sweden 10 or fewer 

France 42 Yes Yes Yes Yes Yes Switzerland —10 or fewer 

Germany f22 Yes Yes Yes No Yes Taiwan 26 No Yes Yes No No 
Hong Kong 10 or fewer Thailand 10 or fewer 

India "140 No Yes Yes Yes Yes Turkey 10 or fewer 

Ireland 10 or fewer Ukraine 10 or fewer 

Israel 10 or fewer United Kingdom’35 Yes Yes Yes Yes No 
Italy 10 or fewer United States 500 Yes Yes Yes Yes Yes 
Japan 131 Yes Yes Yes Yes Yes Vietnam 10 or fewer 


Korea, Republic of 50 Yes Yes Yes Yes Yes 
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Organization Type 


Based on Primary Standard Industrial Classification (SIC) 


C od e ; Wholesale Trade Transportation, 
Retail Trade 1.0% Communication, Electric, Gas 
_ 0.4% \ and Sanitary Senices 
Mining : / Lf 3.2% Finance, Insurance and Real 


Ot ie ; fe = Estate 
— \ ee 5.9% 


Public Administration 
(Including Defense) 


10.1% 
Fabricated Metal Products 
- 0.4% 
Business Services 
40.8% ; . 
Primary Metal Industries 
— 1.0% 
Oe Industrial Machinery And 
55.4% — Equipment 
— 2.2% 


Manufacturing 
23.8% 


7 Electronic & Other Electric 
i Equipment 
5.1% 


Instruments And Related 
Products 
5.9% 


Transportation Equipment 


/ j - Health Services 9.1% 
Engineering & Management = / / 0.6% 
Services — | 
13.3% Other Services 


0.8% 
Based on 505organizations reporting SIC code. For more information visit: 


http://www.osha.gov/oshstats/sicser.html 
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Reporting Organizational Types 


Commercial/In-house 


64.0% 
Contractor for 
Military/Government 
Military/Government 
Agency 
t t | 
600 700 800 


Number of Organizations 


Based on 878 organizations 9/30/05 
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Countries where Appraisals have been 


Performed and Reported to the SEI 


Argentina Australia Austria Belarus Belgium 
Chile China Colombia Czech Republic Denmark 
France Germany Hong Kong India Ireland 
Japan Korea, Republic of Latvia Malaysia Mauritius 
New Zealand Philippines Portugal Russia Singapore 
Spain Sweden Switzerland Taiwan Thailand 
United Kingdom United States Vietnam 


Red country name: New additions since October 2005 


Brazil 
Egypt 
Israel 
Mexico 
Slovakia 
Turkey 


Canada 
Finland 

Italy 
Netherlands 
South Africa 
Ukraine 
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Maturity Profile by All Reporting 
USA and Non-USA Organizations 


100% 
90% + @ USA: 100 % = 420 
80% —- @ Non-USA: 100 % = 686 


60% -- 


40% + 151 239 


Organizations 
o1 
S 
& 


% of 


213 


Not Given Initial Managed Defined Quantitatively Optimizing 
Managed 


Based 0n420 USA organizations and686 Non-USA organizations 
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High-Value Benefits 


David M. Raffo, Ph.D. 
Quanitel, Inc. 

Software Engineering Institute 
Portland State University 


raffod@pdx.edu 
c) 503-939-1720 
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Agenda: Part | 


Introduction: What is Process Simulation? 


Motivation: What can be done with Process 
Simulation Models? 

Examples of High Value Add Ways the Process 
Simulation Can be applied within an organization 


Wrap-Up/ Conclusions 
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What Is a Simulation Model? 


A simulation model is a computerized model 
(not a maturity model) designed to display 
Significant features of the dynamic system it 
represents. 

Simulations are generally employed when 


= behavior over time is of particular interest or 
significance, and 


a the economics or logistics of manipulating the 
system being modeled are prohibitive 
Common purposes of simulation models are: 
a to provide a basis for experimentation, 
= to predict behavior, 
a to answer “what if’ questions, 
= to teach about the system being modeled. 


‘Wnat ie Proce Quante 
Wh at IS Pro CeSS Simulation? Quantitative Intelligence 


Process simulation models focus on the dynamics 
of systems development, maintenance and 
acquisition projects 
They represent the process 
m as Currently implemented (as-is, as-practiced, as- 
documented), or 
= as planned for future implementation (to-be) 


Simulation Features 

a» Use Graphical interfaces 

a Utilizes actual data/ metrics 

=» Predict performance 
Supports “What if’ Analyses 
Support business case analyses 
=» Reduces risk 
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Applying Process Simulation = High Value Add 


Evaluate Strategic Issues 
Quality Assurance, V&V and IV&V Strategy for a 
project 
Globally Distributed Software Development 


Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and make better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 
Architect, Design, and Document Processes 
Estimate Project Costs from the Bottom Up 

Manage Projects Quantitatively 

Train Project Managers 
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Applying Process Simulation = High Value Add 


= Evaluate Strategic Issues 
[1 Quality Assurance, V&V, and IV&V Strategy fora 
project 
(1 Globally Distributed SW Development 
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NASA Model — IEEE 12207 Software 


Paciath wenn 
Development 


Open Interface Screen Run Model 


Results: 
Ne] [eS eyAlete] | MIE] Tt Sie] ele] | ey Te SpcPeyeyal se] | |-—vsee ——tsee sis Dev nt 


~ < 
QA Staff Utilization ie VV Staff Utilization “~ Effort 5 3.3 


20 3 8 0,03 
: Bs an Rework Effort 7 46 
—_ 
a. ik 
Q - lo ht) i) 


Duration 5 1 
Avg. Duration i 0.9 
‘0 
8197.712 8197.712 ° $197,712 


Inj. Defects f 151 
Det. Defects 5994. 137 
Cor. Defects 5978. 137 
Latent Defects 


ON One wna oO 


3: 
ime ime Time 
nem Y2 Utilization ~~ Numb nner Y2 Utilization «mm Numbel — Requests asm YZ Utilization wm Numb 


iE 
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IV&V Layer — Select Criticality Levels for 
IV&V Techniques using pull-down menus 


J Notebook - IEEE12207_Baseline.mox 


IV&V Profile: Silap-Unmanned Load|V&V Profile: (sd 
Save IV&V Profile| Delete Iv&v Profile: [____ 


Concept Verification Requirements Verification Design Verification Code Verification Validation 
IV&V Technique Error Potential Error Potential Error Potential Error Potential Error Potential 


4.4 Management and Planning of 
Independent Verification and Validation 


1.2 Issue and Risk Tracking 
1.3. Final Report Generation 


1.4 IV&V Tool Support 

1.5 Management and Technical Review 
Support 

1.6 Criticality Analysis 


4.7. Identify Process Improvement 
Opportunities in the Conduct of IV&V 


Software Architecture Assessment 


System Requirements Review 


Concept Document Evaluation 


Software/User Requirements 
Allocation Analysis 


Traceability Analysis 


Traceability Analysis — Requirements 


Software Requirements Evaluation 


Interface Analysis — Requirements 
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IV&V Layer — Select Criticality Levels for 
IV&V Techniques using pull-down menus 


4] Notebook - IEEE12207_Baseline.mox \- 


Software/User Requirements 
Allocation Analysis 


Traceability Analysis 


Traceability Analysis - Requirements 
Software Requirements Evaluation 
Interface Analysis - Requirements 
System test Plan Analysis 


Acceptance Test Plan Analysis 
Timing and Sizing Analysis 


Traceability Analysis - Design 
Software Design Evaluation 


Interface Analysis Design 
Software FQT Plan Analysis 


Software Integration Test Plan 
Analysis 
Database Analysis 


4.7 | Component Test Plan Analysis 
Traceability Analysis - Code 


Source Code and Documentation 
Evaluation 
.3| Interface Analysis - Code 


Traceability Analysis - Test 


Le = Quantel 
Impact of IV&V at Different POINts In 2xzntiatve imetigence 


the Development Process 


Result Comparison 


Total Effort} Rework Effort} Duration Mean Corrected Defects Latent Defects 
Case |Configuration Mean Mean Mean 
(Person Months)} (Person Months) 


346.26 201.65 58.42 6,038.26 629.48 


% Improvement Compared to the Baseline 


Total Effort} Rework Effort Duration Corrected Defects Latent Defects 
Case Configuration st Mean Mean Mean Mean 


| [Baseline ae (es ee 
|. 2|TV&vV at Validation a= -4.51%* -2.63%* +1.25% +8.79%* 


IV&V at Code +3.50%* +6.01%* +1.77% +1.60% +8.90%* 
IV&V at Design +5.29%* +9.09%* +3.17%* +1.41% +7.66%* 
IV&V at Requirements +5.62%* +9.64%* +3.46%* +0.67% +4,68%* 


:.  — ae 
GSD Model Structure 


Workforce Available Workforce Available 


: ae Total Tasks Completed j ey 
Site-specific SD Site-specific SD 


Productivit cl Productivit 
i“ Interaction Effect y 


uonesol[V dN 


Defect Rates Defect Rates . 
Site 2 


MP Allocation 


Site 1 


Site-specific DES 


Site-specific DES Tasks Completed 


HR = Human Resource MP = Manpower Allocation PD = Productivity QA = Quality Assurance 
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Interaction Effects (IE) 


= Capture the impact of GSD factors on 
productivity and defect generation rate. 


= Interaction effect on productivity rate 


Productivity 

(Single-site) 
Difference - 1 
| 


+ 
+ 
Coordination Productivity ee Productivity 
Efficiency Multiplier (Multi-site) 
+ 


. Communication 
Frequency 
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Evaluate Process Tradeoffs 


= Task Allocation Strategy Alternatives 


RE DES CODE TEST RWK i -Si 


Module-based 


: RE TEST RWK 
stea [_REQ Eom eae 


Phase-based 


Follow-the-sun 
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Ideal Situation 


100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 

0% 


Percent Completed 


0 50 100 150 200 250 300 Days 


j= Single site ———Follow-the-Sun «= <Phase-based = Module-based 


= Duration when using follow-the-sun is 70% of the 
time it takes using single-site 
= Module-based took a little longer than follow-the-sun 
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Real World Situation 


Percent Completed 


0 50 100 150 200 250 300 350 400 Pays 


«=== Single site ——=Follow-the-Sun «-Phase-based ©=Module-based 


= Follow-the-sun took about 37% longer than single- 
site 
= Module-based Is the shortest 
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Key Questions GSD Models Can Address 


Impact of moving to multi-site development 

Impact of adding a new development site 

Task allocation strategy 

Multi-site QA strategy 

Impact of different development sites using different 

processes, people and technology 

= Deals with issues due to cultural, language, time 
zone, productivity and cost differences 

=» Examines impact of personnel turnover and skills 

development 
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Applications for System Acquisition 


= Can assess impact of using prime with 
collection of subcontractors at different sites 
(.e. software acquisition model) 


= Impact of short funding government projects 
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Applying Process Simulation = High Value Add 


= Evaluate Strategic Issues 
Quality Assurance/ V&V Strategy for a project 
IV&V Strategy 
Globally Distributed Software Development 


= Assess the Costs and Benefits of Applying New 
Tools and Technologies 


: Quantel 
Cost/Benefit of New Technologies —_ 2znitaive inetigence 


CM a anertte ies Changes to 
: 10,885 modules in 
level0 Peli, pelea in? Be ecanG 
13 
a KCl: ee in ’C++’ Modified 
Fs { KC2: 523 functions in “C++” Process 


Results Showing Impact on NASA Projects 


30 detectors, sorted by effort 


Changes to Process 


Steps 
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General Business Case Questions 


a What is the impact of applying new tools and 
technologies? 


a What is the economic benefit or value of the tool or 
technology? What is the Return on Investment? 


a When is it useful and when might it be useless? 


= Under what conditions does the tool or technology 
perform best? 


a What performance standards does the tool need to 
achieve in order to have a positive return? 


= Are there better ways to apply the tool? 
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Applying Process Simulation = High Value Add 


= Evaluate Strategic Issues 
Quality Assurance/ V&V Strategy for a project 
IV&V Strategy 


= Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and Make Better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 


Incremental Development Model 


—— = 


41 SEI Baseline Model. mox 


gg tl = 


Effort 
Rework Effort 
Duration 
Avg. Duration 
Injected Defect 
Detected Defect 
Corrected Defer 
Latent Defects 


ON Own kw N eo 


Entities 
Exited 


Run Model 
Requirements Creeps; [Excluded 7 


Requirements Elicitation Type; |R High Level Design Type: |High Level Design 
reuters peste pe: REUETTSHRGROT ———]__ helen mec 
Requirements Check Toe: [We] Low Level Design specton ype 
Requirements Inspection Type: Code inspection Type: 


Mean 


TOT.T477 418179 


119507.04842493 
28716.842285479 
4244, 1503754571 
3842.840215415 

7003.6032181635 
7003.6032181635 
6640.0895563289 
383,51286 183864 
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Benefits of Process Simulation 


Projected 
LOxoy~1 m0) f 
Total Rework Revenue Rework 
Saas Saat Project delta due Effort for 
(PM) Dev Devel Duration to oct Field Impleme 
ei Dev Defects (Calendar pyration Injected Corrected Escapted Defects ntation 
Option wk (PM) Months) Change Defects Defects Defects py)  Costs($) 


= Implement QFD a $0.00 1150 1020 $100,000 | $165,145 15% 


$ 80,000 | $289,674 
$0.00] -$378,043}  -129% 


a QuARS Tool $ 300,000 1150 1090 


ae ie VOC a a $ 100,000 1150 1050 $120,000 | $185,231 29% 
a 
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Applying Process Simulation = High Value Add 


= Evaluate Strategic Issues 
Quality Assurance/ V&V Strategy for a project 
IV&V Strategy 
Globally Distributed Software Development 


= Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and Make Better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 
Architect, Design, and Documenting Processes 
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Architect, Design and Document Processes 


Process Simulation Model 


Generic Process Model Blocks 


== 


Generalized Process Components 


Reqi: Use Case Analysis 
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Applying Process Simulation = High Value Add 


Evaluate Strategic Issues 
Quality Assurance/ V&V Strategy for a project 
IV&V Strategy 
Globally Distributed Software Development 


Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and Make Better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 
Architect, Design, and Documenting Processes 
Estimate Project Cost from the Bottom-Up 
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Using Process Simulation to perform early 

stage project cost estimation 

= Study Conducted by Mizell, in the Engineering 
Assessment Directorate at KSC 


= Applied Process Simulation to provide bottom-up 
cost and schedule estimates at multiple stages of 
the project (i.e. from Concept of Operations forward) 
a Utilized real project data from KSC and SEL 
= Developed estimates that incorporated effects for 
Incremental Spiral processes 
Impact of short funding projects 
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NASA Model — Incremental Spiral 
Lifecycle 


PATT 


| Repeat for Each Spiral | 


Add Lessons 


Add Risk Learned 


Assessment 


Open Interface Screen Run Model 
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Adapted from Mizell, 2006 
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Accomplishments of Mizell’s Research 


= Methodology to use simulation to provide interval 
estimates 


= Developed probability distributions for size, 
er and defects using organization specific 
ata 


= Provided confidence intervals for project estimates 


= Combined system dynamics model with DES process 
model to analyze effects of turnover on project effort 
and duration 


= Adapted incremental spiral process model 
= Complete NASA project case study 


© Mizell, 2006 
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Process Models Used 


mw IEEE 12207 being used by NASA IV&V 
= Adapted for incremental development 
= Adapted for spiral development 


= Incorporated system dynamics portions into 
model 


= Model development supported by Quantel 
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Applying Process Simulation = High Value Add 


Evaluate Strategic Issues 
Quality Assurance/ V&V Strategy for a project 
IV&V Strategy 
Globally Distributed Software Development 


Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and Make Better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 
Architect, Design, and Documenting Processes 
Estimate Project Costs from the Bottom-Up 

Manage Processes Quantitatively 
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PROMPT: Quantitative Project Management Framework 


NASA SW Project and IV&V 


Project and IV&V Data 
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SW Process Simulation Model 
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Outcome Based Control Limits 
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PROMPT Control and Feedback Loop 


Metrics 
Repository 


S corrective 


action 
I necessary? 
Updated Model ELMaly ‘ ) 
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n Control? 


Secondary 


SW Process Loop 
imulation Model 


Outcome Based Control Limits 
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Applying Process Simulation = High Value Add 


Evaluate Strategic Issues 
Quality Assurance/ V&V Strategy for a project 
IV&V Strategy 
Globally Distributed Software Development 


Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and Make Better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 
Architect, Design, and Documenting Processes 
Estimate Project Costs from the Bottom-Up 

Manage Processes Quantitatively 

Train Project Managers 
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Teaching Software Project Management through 
Modeling (Navarro, et. al., 2006) 


aul WoRCAS | | 
BRB RAe SS 


—s ... eee Current Activities: 
: oi 
| | 
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Benefits of Process Simulation 


Decision Support and Tradeoff Analysis 
Sensitivity Analysis — “What if” 

Supports Industry Certification and process 
improvement programs including CMMI, 
Six Sigma, and others 

Benchmarking 

Design and Define Processes/Metrics 
Bring Lessons Learned Repositories Alive 
Can save cost, effort, and expertise 

Many ways to achieve High Value-Add by 
using process simulation 
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Bottom-Line 


m™ Process Simulation can make an impact on your 
business. 


Improving QA strategies (defect containment, COQ, 
COPQ) 


Achieving higher CMMI Levels (Fulfilling CMMI L4) 
Implementing 6 Sigma practices 
Adopting new technologies 
Plan/replan projects 
Bottom-up cost estimation 
=» Enables an organization to adapt to change and 
improve processes more quickly — beating the 
competition, win contracts 
=m Enables an organization to design processes better, 
train employees, implement more quickly = better 
performance, higher quality, faster 


The End 
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High-Value Benefits 


David M. Raffo, Ph.D. 
Quantel, Inc. 

Software Engineering Institute 
Portland State University 
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Agenda: Part II 


a Overview of Simulation Types 
= Process Tradeoff Analysis Method 
Data 
Model Templates 
Model Database 
Analysis of Results 
= Incremental Model tour 
= Conclusions 
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Alternative Process Simulation Approaches 


= Modeling Paradigms ¢ Tools 
Knowledge-Based — Arena 
systems — ProModel 
Agent Based — Extend 
State-Based — Stella 
Discrete Event — VenSim 
System Dynamics — Research tools 
Hybrid 


e Conferences 


=» Research Outlets — Winter Simulation 


-~ 7 . Conference 
oftware Process: _ SPW/ ProSim 
Improvement and 
Practice — SEPG 
—SSTC 


Journal of Systems and 
Software 


te Quantel 


Quantitative Intelligence 


Alternative Process Simulation Approaches 


= Knowledge Based Systems 
Person-in-the loop 
Fine level of granularity 
Supports process enactment 
= Agent Based Systems 
Fine level of granularity 
Supports detailed work interactions 
= State Based Systems 


Captures flow of control (work activities, 
parallelism) well 


Multi-view graphical representations 


Difficult to capture task, work package and 
resource details 
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Alternative Process Simulation Approaches 
=m Discrete Event Simulation 


Able to represent richness of processes, work 
packages and resources 


Good for modeling quantitative process performance 
Good tool support 
a System Dynamics 
Captures feedback well 
Often used for high level qualitative issues 
m Hybrid 
Captures best aspects of Discrete Event and System 
Dynamics 
Models are complex 


Being used to predict performance of multi-site 
development 
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Common Applications of Each Approach 


STRAT PLAN MGMT IMPR UNDR TRAIN 


KBS Xx X | 
Agent Based XxX xX 
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System Dynamic Xx X x Xx Xx | 
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INo. of CSCIs 8 
CSCI names: C&DH Guidance & MEPS Ground 
Estimated SLOC 


Reuse Re-eng New Lang 


25000 75000 
25000 75000 


eOrganizational 


eSite and Project “> 


¢ Industry Standard 


SW Process Simulation Model 


DIVINER LAMP LOLA 


Ptotals 
120000 
120000 


LROC 


IVVTotals 
150000 
150000 


EP cP 


3 


Project 
Database 


“SIEEE 12207 PATTZimox 


Development Project Statistics 


3432135 
1434.76 


1056858 
21746 


i 


53 ARS RS 
151191 
BIN 
1434.75 
Roma 


O47317 
29802 
TOSARSS 
21746 
904353 


20273 


te Quantel 


Quantitative Intelligence 


Where does the data come from? (1 of 2) 


Input data are used to predict the performance 
measures. 
Can be derived from the organization 

= Current baseline 

=» Exemplary projects 

= Pilot data 

Can also be derived from 

=» Expert opinion 

= Industry data from comparable organizations 


Best judgments to describe the state of your 
organization 
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Input Data @ of2) 


=» Examples: 
process documents and assessments 
amount of incoming work 
effort based on size (and/ or other factors) 
defect detection efficiency 
effort for rework based on size and number of defects 
defect injection, detection and removal rates 
decision point outcomes; number of rework cycles 
hiring rate; staff turnover rate 
personnel capability and motivation, over time 
resource constraints 
frequency of product version releases 
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Creating Process Simulation Models 


Process Simulation Model 


FS Development Unit Test Release to 
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Life Cycle Model Templates Generic Process Model Blocks 


REQ {DES} IMP} ALES T} +CUST 
Be a 


e IEEE 12207 ° 
. : Generalized Process Components Deve epinen 
¢ Spiral e Inspection 


¢ Incremental Req1: Use Case Analysis ¢ Testing 
e Product Line e Rework 
e Rapid Prototyping Eo a) IV&V 


«Joint Reviews 
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Quantitative Intelligence 
Customizing PATT 


Multiple block types implement SW development techniques 


product and inject defects 
Inspection blocks detect defects 


Testing blocks detect defects 


a|(==\fe ™ Rework blocks correct and inject 
defects 


= Joint Review blocks detect and 
correct defects. 


3 = m@ IV&V blocks detect defects. 
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Project Data Base 


= Inputs 


Size, productivity, error potential, consequence, defect 
injection, detection, and correction rates, cost, duration, 
etc. 


= Outputs 
Customizable reporting 
All levels - Project, phase, activity levels 
Costs reported using COQ format 
Defect containment statistics 
Special reports for IV&V 


SS 
Project Database 


f= |FEE1?207Data : Database (Access 2000 file format) 


| | 45 fe [| # 
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Create table in Design view 
Create table by using wizard 
Create table by entering data 
Entity_Info 

Entity_List 

Fan_Out 
Ivv_Activity_Statistics 
Ivv_Input 

Ivv_Phase_Data 
Ivv_Phase_Statistics 
IVV_Profiles 

Ivv_Statistics 


J 
| 
gj 
3 
ga 
ee 
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Ivv_Step Data 
Ivyv_Totals 
Parameter_Information 
Parameter_Names 
Project_Activity_Efforts 
Project_Actvity_Statistics 
Project_Data 
Project_Level_Info 
Project_Phase Data 
Project_Phase Statistics 
Project Statistics 
Project Step Data 
Project_Totals 


4) Create report in Design view 


| Create report by using wizard 


J) Development Activity Defect Statistics 


i Development Activity Effort/Duration Statistics 


J Development Activity Total Effort/Ouration Statistics 
J Development Phase Defect Statistics 

J Development Phase Effort/Duration Statistics 

9 Development Phase Total Etfort/Duration Statistics 


Development Project Statistics 
J = Development Project Statistics (Includes Total Effort} 


{9 Development Project Totals (Includes Total Effort} 
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IVaV Activity Statistics 

IVaV Phase Statistics 

IVAV Project Statistics 

IVY Project Totals 

Project Activity Effort Statistics 

Project Effort Statistics 

Project Effort Totals 

Project Phase Effort Statistics 

Single Development Phase Defect Statistics 

Single Development Phase Total Effort/Duration Statistics 
Single Phase Development Activity Defect Statistics 
Single Phase Development Activity Tot Effort/Duration Statistics 
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Development Project Total Effort/Duration Statistics 


Project Name: IEEEI 2207 SW Develormert Process Model 
Ru Set i Rework Effors 


30,541.63 
0.00 


30,541.63 
0.00 
33,233.51 
0.00 
33,736.25 
0.00 
30,541.63 
0.00 


38,632.66 30,541.63 
0.00 0.00 
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PATT Architecture 
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Benefits of Process Simulation 


Projected 
LOxoy~1 m0) f 
Total Rework Revenue Rework 
Saas Saat Project delta due Effort for 
(PM) Dev Devel Duration to oct Field Impleme 
ei Dev Defects (Calendar pyration Injected Corrected Escapted Defects ntation 
Option wk (PM) Months) Change Defects Defects Defects py)  Costs($) 


= Implement QFD a $0.00 1150 1020 $100,000 | $165,145 15% 


$ 80,000 | $289,674 
$0.00] -$378,043}  -129% 


a QuARS Tool $ 300,000 1150 1090 


ae ie VOC a a $ 100,000 1150 1050 $120,000 | $185,231 29% 
a 
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Computations and Tradeoffs 


a Sensitivity Analysis 

= Design of Experiments 

=» Business Case — ROI, NPV 
= Methods in use and available 
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Demonstration of the Incremental Model 


Sr + eS 


4 SEI Baseline Model.mox 


Requirements Inspection Type: |Requirements Inspection - Baseline - 
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Run Model 
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Simulation User-Levels 


= Level 1: Manager - Runs simulations based on pre- 
determined options 

= Level 2: Analyst - Able to add or change the process 
to study the impact of process changes 

= Level 3: Expert - Able to create new models from 
scratch 

= Level 4: Developer - Able to program new block 
and/or modify the logic of existing blocks, as allow 
by the security model 


= Level 5: Originator — Establishes security model 
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Process Tradeoff Analysis Method (PTAM) 


Based on extensive research into Software Process 
Modeling conducted in academia, SEI and industry. 


Graphical user interface and models software 
processes 


Integrates SEI methods to define processes and 
supports CMMI PAs (CMMI L4 QPM) 

Supports Industry Certification Programs including 
CMMI, Six Sigma, and others 

Benchmarking 


Integrates metrics related to cost, quality, and schedule 
into understandable project performance picture. 


Predicts project-level impacts of process 
improvements in terms of cost, quality and cycle time 
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Process Tradeoff Analysis Method (PTAM) 


Support business case analysis of process 
decisions - ROI, NPV and quantitatively 
assessing risk. 


Reduces risk associated with process changes 
by predicting the probability of improvement 
Saves time, effort and expertise over other 
methods 
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Applying Process Simulation = High Value Add 


Evaluate Strategic Issues 
Quality Assurance, V&V and IV&V Strategy for a 
project 
Globally Distributed Software Development 


Assess the Costs and Benefits of Applying New Tools 
and Technologies 


Plan Processes and make better Tradeoff Decisions 
Evaluate Process Improvement Opportunities 
Architect, Design, and Document Processes 
Estimate Project Costs from the Bottom Up 

Manage Projects Quantitatively 

Train Project Managers 

Process Simulation can make a positive 
impact on your business! 


The End 
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Agenda 


= An Overview of Levels 4 and 5 


—/ = New Behaviors 
a = Benefits — Project, Organizational, Customer 
| = Making the Business Case 


= Understanding the CMMI Process Areas 
= Organizational Process Performance 
=" Quantitative Project Management 
= Causal Analysis & Resolution 
"= Organizational Innovation & Deployment 


= Strategies for Adoption 
= Lessons Learned 
= References 


SM SCAMPI, SCAMPI Lead Appraiser, and SEI are service marks of Carnegie Mellon University. 
® Capability Maturity Model Integration and CMMI are registered in the U.S. Patent & Trademark Office. 
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‘Project 


5 Optimizing 


Quantitative 
management 


4 Quantitatively 
Managed 


3 Defined 


Proactive 
management 


Reactive mgmt. 
| (plan, track, and 
| correct) 


2 Managed 


1 Performed 


Management Styles in the CMMI 


Causal Analysis and Resolution 
Organizational Innovation and Deployment 


Quantitative Project Management 
Organizational Process Performance 


Requirements Development 
Technical Solution 

Product Integration 

Verification 

Validation 

Organizational Process Focus 
Organizational Process Definition 
Organizational Training 

Risk Management 

Integrated Project Management (for IPPD*) 
Integrated Teaming* 

Integrated Supplier Management** 
Decision Analysis and Resolution 


Organizational Environment for Integration* 


Requirements Management 

Project Planning 

Project Monitoring and Control 

Supplier Agreement Management 
Measurement and Analysis 

Process and Product Quality Assurance 
Configuration Management 


Organizational 


Quantitative 
improvement 


Qualitative 
improvement 
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Exercise - 
What ts Quantitative Management? 


= Suppose your project = What would you 


w 4 conducted several peer 
at reviews of similar code, and ss et iets a 
3 analyzed the results terms of parr 
, =» Mean = 7.8 defects/KSLOC KSLOC? 
; » +30 = 11.60 defects/KSLOC What id 
a m at would you 
30 = 4.001 defects/KSLOC think if a review 
- resulted in 10 
UCL=11.60 defects/KSLOC? 
> = 3 defects/KSLOC? 
= 9 
5 8 Mean=7.8 
‘=: 7 
2 
— B 6 
el 7 5 
ea 4 LCL=4.001 
3 
0 5 10 15 
Observation Number NORTHROP GRUMMAN 
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Exercise - 
What is Required for Quantitative Management? 


*Z = What is needed to develop the = The process has to be 
—" statistical characterization of stable (predictable) 
Fs a process? = Process must be 
gt consistently performed 


= Complex processes may 
need to be stratified 
(Separated into simpler 
8 OS processes) 


10 = There has to be enough 
data points to statistically 
characterize the process 


=» Processes must occur 


9 

8 Mean=7.8 
7 
6 frequently within a similar 
5 
4 
3 


Individual Value 


context (project or 
LCL=4.001 organization) 


@) 5 10 15 
Observation Number 


a 
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What Is a Data Distribution? 


s = The data distribution 
— , gives a frequency of 

occurrence of each 
value in a data set 


Fe i a Process Example: Closing a change request 
20 - =" Once we characterize 
the data distribution, we 
C15 - can predict future 
gv values or assigna 
g probability to any 
ial specific value 
5- = Data distributions may 
represent population or 
0 process data 
4 7 10 13 16 


Hours to Close 


=" When applied to 
process data - assumes 
statistical stability 
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The Normal Distribution 


=" Commonly called a bell curve; 
formally called a Gaussian 
distribution 


= Applies to continuous data (e.g. 
time, weight, distance) 


=" Characteristic shape: 


= Mean, mode, and median are 
all the same value 


= Standard deviation is 
independent of the mean 


= Normal distributions are nota 
law, but they are often observed 
in natural data 


Frequency of Occurence 


-16-12 -8 -4 0 4 8 12 16 20 
Data Value 
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Characteristics of a Normal Curve 


o=1 standard f 
deviation 


Once we understand what 
kind of distribution our 
data comes from, we can 
predict future values or 
assign a probability to 
any specific point 


Probability x will occur 
Probability a value < x 
Probability a value > x 


Probability a value will 
occur between x and y 
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The Rayleigh Distribution... 


= A skewed distribution 


= Used to model effort or defect 
density in software 
development where the 


x ov : 
: = 250 abscissa represents 
= development phase if they are 
g 500 roughly the same duration 
‘OS 
o 150 
Cc 
oO 
= 
© 100 
uo 


50 


Sata Value 
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Example - Defect Detection 


| AWIPS Rel 4.2 DRs 
400 
300 
200 
- lm DRs 
fa al : 
a — Rayleigh Model 
100 
0 
VTERGMNMIAS4 56770 
oe a9 90 oo 09 90 
Date 
“Business Value and Customer Benefits Derived from High Maturity”, NORTHROP GRUMMAN 


Al Pflugrad, CMMI Technology Conference and User Group, 2002 
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What Is a Control Chart? 


= A time-ordered plot of process 
data points with a centerline 
based on the average and 
control limits that bound the 
expected range of variation 


= Control charts are one of the 
most useful quantitative tools 
for understanding variation 
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What Are the Key Features of a Control Chart? 


Individual Value 


Lower 
Control 
Limit 


A? 


Upper ————_——_——_—_ | UCL=11.60 


Control 
Limit 
Process 
“Average” 
Mean=7.8 
Individual 
data points 
LCL=4.001 


5 10 Time 
Observation Number ordered 
X-aXlS 
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There are Many Types of Control Charts 


U Chart of Defect Detected in Requirements Definition 


— / 0.10 
& Lr _ | ve =0.09633 
3 0.09 
“4 rw} 
6 
2 
§ 0.08 os 
. U=0.07825 
2 
Ww) 
% 0.07 
2 . 


0.06} po | EL =0.06017 


1 3 5 7 9 11 13 15 17 #19 21 23 25 
ID 


Tests performed with unequal sample sizes 


NORTHROP GRUMMAN 


13 Copyright 2005 Northrop Grumman Corporation 


What is Special Cause and Common Cause 
Variation? 


Individual Value 


15 


Individual Value 


14 


UCL=11.60 


LCL=4.001 


Veves Nicaea 


5 10 15 
Observation Number 


UCL=13.36 


Mean=7.467 


LCL=1.578 


| | | 
5 10 15 


Observation Number 


=" Common Cause Variation 


= Routine variation that comes 
from within the process 


=" Caused by the natural variation 
in the process 


= Predictable (stable) within a 
range 


= Special Cause Variation 


= Assignable variation that comes 
from outside the process 


= Caused by a unexpected 
variation in the process 


= Unpredictable 
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What Is a Stab/e (Predictable) Process? 


~~ " U Chart of Defects Detected in Requirements Definition 
P ' aL 7_ _| 0.09833 


0.09 


0.08 - 
U=0.07825 


All data points within 
0.07 the control limits. No 
Signals of special 
cause variation. 


0.06 po | 10.0017 


1 3 5 7 9 11 13 15 17 19 21 23 25 


Defects per Line of Code 
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What if the Process I sn’t Stable? 


= You may be able to explain 
— ~ out of limit points by 
= observing that they are due 
: to an variation in the 
a process 
u = E.g., peer review held on 
Friday afternoon 
= You can eliminate the 
points from the data, if 
they are not part of the 
process you are trying to 
predict 


N 
ol 


N 
o 


Defects per component 
- 
ol 


= You may be able to stratify 
the data by an attribute of 
the process or attribute of 
the corresponding work 
product 
| = E.g., different styles of 
os peer reviews, peer 
| reviews of different types 
of work products 


is eas i a A 
1 1 21 #31 41 «51 661) 71 
Component # 


Defects per component 


° a 


wg a cL : 
1 11 21 31 41 51 61 71 
Component # 


° a 
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Hearing Voices 


= Voice of the process 
— = the natural bounds of process performance 


ee = Voice of the customer 
rd = the goals established for the product/process performance 


= Voice of the business 
= process performance needed to be competitive 


= Process capability may be determined for the 
= Organization 
= Product line 
= Project 
= Individual 


= Typically, the higher the level of analysis, the greater the variation 


NORTHROP GRUMMAN 
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Common Challenges for Engineering 


= Data are often discrete rather 
than continuous, e.g., defects 


= Observations often are scarce 
= Processes are aperiodic 


= Size of the the object often 
varies, e.g., software module 


= Data distributions may not be 
normal 


NORTHROP GRUMMAN 
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How Do! Address These Challenges? 


= Employ control chart types that 
wo specifically deal with discrete data 
" distributions, e.g., u-charts and p- 
87 charts 


= Use control charts that compensate 
for widely variable areas of 
opportunity 


= Transform non-normal continuous 
data to normal data before 
constructing a control chart 


= Cross check control charts with 
hypothesis tests where few data 
points exist 


| NORTHROP GRUMMAN 
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Typical Choices in Industry 


= Most customers care about: 
= Delivered defects 
= Cost and schedule == 


Defect Detection Profile 


= So organizations try to 
predict: 
=" Defects found throughout 
the lifecycle 


= Effectiveness of peer 
reviews, testing 


= Cost achieved/actual a 
(Cost Performance Index — i —-= 
C Pp | ) _ Req'mts ) Design ) Code ) Unit Test ) Integrate . Sys Test Del 90 Days 
= Schedule achieved/actual 
(Schedule Performance 


Index — SPI) 


DAII Projects 
iNew Process 


Defects/KSLOC 
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How Can High Maturity Help? 


=" By measuring both the mean 

; and variation, the project/ 

= f organization can assess the full 
% impact of an “improvement” 


& = Can focus on reducing the 

a variation (making the process 

more predictable) 15 
= Train people on the process 
=" Create procedures/checklists 
= Strengthen process audits 


ra 
oO 


= Can focus on increasing the 
mean (e.g., increase 
effectiveness, efficiency, etc.) 


= Train people 

= Create checklists 

= Reduce waste and re-work 

=" Replicate best practices from 
other projects 0 5 10 15 


Observation Number 
= Can do both 


Individual Value 
ol 


oO 
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What Can a Level 4 Organization/ Project Do? 


*Z4 «= Both can: 
<7 = Determine whether processes are behaving consistently or have stable 
r trends (i.e., are predictable) 
rh ‘. = Eliminate special causes of variation 


= Identify processes that show unusual (e.g., unpredictable) behavior 
= Identify the implementation of a process which performs best 


= The organization can: 


= Characterize the process performance of the organization’s standard 
process 


= Develop models to predict process performance 


= The organization’s projects can: 
= Use this data to decide how to tailor the organization’s standard process 


= Statistically manage selected subprocesses to achieve project quality and 
process objectives 


NORTHROP GRUMMAN 
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What Does Level 5 Add? 


Organizational Innovation & 
Deployment 


= Goals are quantitative (e.g., 
reduce variation by X%, reduce 
mean by Y%) 


= Incremental improvements 


= Innovative improvements - cause 
a major shift in process 
capability 


= Potential improvements are 
analyzed to estimate costs and 
| Organizational Process Focus impacts (benefits) 


= Goals are qualitative (e.g., get = Improvements are piloted to 
better) ensure success 


= The effects of the improvements = Improvements are measured in 
are not estimated or measured terms of variation and mean 
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24 


How Does Level 4 & 5 Benefit the Customer? 


Organizational process 
performance 


Quantitative project 
management 


Organizational innovation 
and deployment 


Causal analysis 


a 


LP, 


“How Does How Does High Maturity Benefit the Customer’, R. Hefner, 
Systems & Software Technology Conference,. 2005 


= More accurate estimates 


= Problem behaviors are 
recognized faster, enabling 
quicker resolution 


= The project fixes the source of A 
defects to prevent future defects 


y 
oy y il 


Better Products and Services Produced Faster And Cheaper 
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Agenda 


= An Overview of Levels 4 and 5 
= New Behaviors 
= Benefits — Project, Organizational, Customer 
= Making the Business Case 


= Understanding the CMMI Process Areas 
= Organizational Process Performance 
=" Quantitative Project Management 
= Causal Analysis & Resolution 
#" Organizational Innovation & Deployment 


= Strategies for Adoption 
= Lessons Learned 
= Links to Six Sigma 
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as 


The Project Manager’s Dilemma at Level 3 


| want to use the organization’s 
standard process, but... 


.. Does it’s performance and quality 
meet my customer’s expectations? 


.. If not, how should | tailor the 
process? 
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What Should the Organization Do to Help the 
Project Manager? 


= Characterize the performance of the organization’s standard 
process Statistically 


= Develop models to help a project manager determine the 
performance they would be likely to get by using the standard 
organizational process model, given their project’s characteristics 


Historical Data Standard Process Baselines & Models 


: Organization 
S Project T y 
customer and 
ss project 
ee (a objectives 
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a7 Project Results Tailored Process 


Establish Performance Baselines and Models 


Baselines and models that characterize the 
expected process performance of the-erganization's 
set of standard processes are established and 

maintained. 


Select Processes 


Select the processes or process elements in the 
organization's set of standard processes that are to 
be included in the organization's process 
performance analyses. 


SP 1.2 Establish Process Performance Measures 


Establish and maintain definitions of the measures 
that are to be included in the organization's process 
performance analyses. 


SP 1.3 Establish Quality and Process-Performance 
Objectives 
Establish and maintain quantitative objectives fq 
quality and process performance for the 
organization. 

SP 1.4 Establish Process Performance Baselines 
Establish and maintain the organization's process 
performance baselines. 

SP 1.5 Establish Process Performance Models 

Establish and maintain the process performance 

models for the organization's set of standard 

processes. 


28 


Organizational Process Performance 


Selected subprocesses, NOT the 


whole process 


Objectives deal with eliminating 
sources of variation, not setting 
“stretch” goals 


The organization meets these 
goals by modifying the standard 
process, not driving the projects 


Baselines characterize the “voice of the 
process”, based on the existing 
historical data 
¢ What is the current mean and 
variation? 


May need to subgroup the data 
Models allow projects to estimate 
their quantitative performance based 


on the historical data of other 
projects executing the process 
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Process Architecture 


Configuration 
Management 


Quality 
Assurance 


Management 


Process Elements 
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Analysis 


Nang 


Decision 
Analysis & 
Resolution 


Alternative Process Elements 
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Quantitative Baselines and Models 


= From this baseline (and model), what would you predict the next 
r peer review to produce in terms of defects/KSLOC? 


a | =" What other models could be developed to help predict? 


| Chart for Defects 
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The Project Manager’s Challenge at Level 4 


! understand the capabilities of the organization’s standard 
process, but... 


.. What are the project’s quality and process performance 
objectives? 


.. How should | tailor the process? 


.. What project subprocesses do 
I need to quantitatively manage? 
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SG1 Quantitatively Manage the Project 


Quantitative Project Management (Goal 1) 


Quality: defect levels of key 
work products or deliverables 


Process: productivity, efficiency, 
effectiveness of the project’s 
processes 


The project is quantitatively managed using 
quality and process-performance object 


SP 1.1 Establish the Project’s Objectives 


Rationale for how the project tailored 
the organization’s standard process, 
in order to meet their quality & 
process performance objectives 
¢ E.g., adding procedures to reduce 
variation 


SP 1.2 Compose the Defined Process 
Select the subprocesses that 


project’s defined process based on historical 
stability and capability data. 


SP 1.3 Select the Subprocesses that Will Be 
Statistically Managed 


Assumes the standard process 
includes subprocesses to select from 
Select the subprocesses of the project's 
defined process that will be statistically 
managed. 


Only some subprocesses selected for 
statistical management 
¢ Need not be the same as those 


| SP 1.4 Manage Project Performance 
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Monitor the project to determine whethe 
project’s objectives for quality and process 
performance will be satisfied, and identify 
corrective action as appropriate. 


selected by the organization, or other 
projects 


Monitoring against the objectives 
established in SP 1.1 
NORTHROP GRUMMAN 
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Outer Loop 


SP 1.1 Establish the Project’s Objectives 


Establish and maintain the project’s quality 
and process performance objectives. 


SP 1.3 Select the Subprocesses that Will 
Be Statistically Managed 


Select the subprocesses of the project's 
defined process that will be statistically 
managed 


SP 1.4 Manage Project Performance 


Monitor the project to determine whether 
the project’s objectives for quality and 
process performance will be satisfied, and 
identify corrective action as appropriate. 
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Selecting Subprocesses to be Statistically Managed 


SP 1.3 Select the Subprocesses that Will 
Be Statistically Managed 


Select the subprocesses of the project's 
defined process that will be statistically 
managed 


= Which processes do you need to be stable 
(predictable) in order to achieve your project’s 
objectives? 
= For these, eliminate special causes, characterize 
the process, and predicatively manage 


= The time needed to perform this practice is long 
and often unpredictable 


= Many processes can not be made predictable 


= Example - objectives for delivered defects 


=" Defect detection (peer review, unit testing, system 
testing) 


" Defect insertion (requirement definition, 
architecture, design, integration) 
NORTHROP GRUMMAN 
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Tailoring 


Organization 
Process Project Project Risk Supplier 
Tailoring Planning Tracking Management Management 
Quality Configuration Data Measurement & 
Assurance Management Management Analysis 


Project 


Process Project Project Risk Supplier 
Tailoring Planning Tracking Management Management 


\ integration & 
a eee, 
Assurance Management Management Analysis 


SP 1.2 Compose the Defined Process 
Select the subprocesses that compose the 


project’s defined process based on 
historical stability and capability data. 
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SG 2 Statistically Manage Subprocess Performance 
The performance of selected subprocesses 
within the project's defined process is 
statistically managed. 


SP 2.1 Select Measures and Analytic Techniques 


Select the measures and analytic techniqdes 
to be used in statistically managing the 
selected subprocesses. 


SP 2.2 Apply Statistical Methods to Understand 
Variation 


Establish and maintain an understanding 
the variation of the selected subprocesses 
using the selected measures and analytic 
techniques. 


SP 2.3 Monitor Performance of the Selected 
Subprocesses 


Monitor the performance of the selected 
subprocesses to determine their capability to 
satisfy their quality and process 
performance objectives, and identify 
corrective action as necessary. 


SP 2.4 Record Statistical Management Data 


Record statistical and quality management 
data in the organization’s measurement 
repository. 
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Quantitative Project Management (Goal 2) 


Type of analysis to be performed 
(e.g., control charts) 


Key is understanding variation 
in the selected subprocesses 
(e.g., be able to compute 
standard deviation), NOT just 
metrics 


Given the stability and variation in 
the subprocesses, will we be able 
to meet our project-level quality 
and process performance 
objectives? 


This data is used to help select 
subprocesses in tailoring (SP 1.2) 
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Inner Loop 


SP 1.1 Establish the Project’s Objectives 


Establish and maintain the project’s quality 


and process performance objectives. 


SP 1.3 Select the Subprocesses that Will 
Be Statistically Managed 

Select the subprocesses of the project's 
defined process that will be statistically 
managed 


SP 1.4 Manage Project Performance 
Monitor the project to determine whether 
the project’s objectives for quality and 
process performance will be satisfied, and 
identify corrective action as appropriate. 


SP 2.2 Apply Statistical Methods to 
Understand Variation 


Establish and maintain an understanding of 
the variation of the selected subprocesses 
using the selected measures and analytic 
techniques. 


SP 2.3 Monitor Performance of the 
Selected Subprocesses 

Monitor the performance of the selected 
subprocesses to determine their capability 
to satisfy their quality and process 
performance objectives, and identify 
corrective action as necessary. 
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Sample Measures for Quantitative Analysis 


= Time between failures 
= Critical resource utilization 


=" Number and severity of defects in the 
released product 


= Number and severity of customer 
complaints concerning the pro-vided 
service 


= Number of defects removed by 
roduct verification activities 
per-haps by type of verification, 
such as peer reviews and testing) 
= Defect escape rates 


=" Number and density of defects by 
severity found during the first year 
following product delivery or start of 
service 


= Cycle time 
= Amount of rework time 


= Requirements volatility (i.e., number 
of requirements changes per phase) 


= Ratios of estimated to measured 
values of the planning parameters 
(e.g., size, cost, and schedule) 


Coverage and efficiency of peer 
reviews (i.e., number/amount of 
products reviewed compared to total, 
number of defects found per hour) 


Test coverage and efficiency (i.e., 
number/amount of products tested 
compared to total, number of defects 
found per hour) 


Effectiveness of training (i.e., percent 
of planned training completed and 
test scores) 


Reliability (i.e., mean time-to-failure 
usually measured during inte-gration 
and systems test) 


Percentage of the total defects 
inserted or found in the different 
phases of the project life cycle 


Percentage of the total effort 
expended in the different phases of 
the project life cycle 


Profile of subprocesses under 
statistical management (i.e., number 
planned to be under statistical 
management, number currently being 
statistically menedeu and number 
that are statistically stable 


Number of special causes of variation 
identified 
NORTHROP GRUMMAN 
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New Questions at Level 4 


i What characteristics of the organizational standard 
@ process would be useful to understand? 


Which subprocesses would be useful to understand, 
for predictive purposes? 


Are these subprocesses predictable (stabilizable)? 


What data should the organization collect? 


To what level of detail should the organizational 
standard process go? 


What differences in project subprocesses are 
permissible? How do they impact the historical data? 
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What Does Level 5 Add to the Project/ 
Organization? 


> 4 " Casual Analysis & Resolution effects 
—_ / = Identify and analyze causes of defects and other -preblerms- 
: =" Take specific actions to remove the causes 


= The project may take actions to prevent the occurrence of those 
types of defects and problems in the future 


= Most projects use CAR-type methods and tools at Level 4, to 
identify and eliminate special cause variations, i.e., to stabilize 
the processes 


= At Level 5, CAR Is used to eliminate common cause variation 
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Causal Analysis & Resolution 


Can apply to any cause and effect 


Determine Causes of Defects relationship, not just defects 
Root causes of defects and other problem ; ! , , 
are systematically determined. a Typically, projects will establish a 


list of potential areas in which to 
apply CAR, and select some from 
that list 


SP 1.1 Select Defect Data for Analysis 


Select the defects and other problems for 
analysis. 


SP 1.2 Analyze Causes 


Perform causal analysis of selected defects 
and other problems and propose actions to 
address them. 


Determine cause and effect 
(e.g., fishbone diagram, 
brainstorming) and potential 
improvement action list 


Select some actions on the list 
to implement 


Address Causes of Defects 
Root causes of defects and other problems 

are systematically addressed to prevent their 
future occurrence. 


Implement the Action Proposals 
Implement the selected action proposals that 
were developed in causal analysis. 
SP 2.2 Evaluate the Effect of Changes 
Evaluate the effect of changes on process 
performance. 
SP 2.3 Record Data 


Record causal analysis and resolution data 
for use across the project and organization. 


Implemented for each selected action 
¢ Evidence will provide samples 


Measures the effect of the 
change 
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Select Improvements 
Process and technology improvements that contribute to 
peeing: quality and process-performance objectives are 
selected. 


Collect and Analyze Improvement Proposals 

Collect and analyze process- and technology- 
improvement proposals. 

SP 1.2 Identify and Analyze Innovations 
Identify and analyze innovative improvements that could 
increase the organization’s quality and process 
performance. 

SP 1.3 Pilot Improvements 
Pilot piocees and technology improvements to select 
which ones to implement. 

SP 1.4 Select Improvements for Deployment 


Select process- and technology-improvement proposals 
for deployment across the organization. 


SG2 Deploy Improvements 


Measurable improvements to the organization's processes 
ene tecniolodigs are continually and systematically 
ed. 


Plan the Deployment 
Establish and maintain the plans for deploying the 
selected process and technology improvements. 
SP 2.2 Manage the Deployment 
Manage the deployment of the selected process and 
technology improvements. 
SP 2.3 Measure Improvement Effects 


Measure the effects of the deployed process and 
technology improvements. 


Organizational I nnovation and Deployment 


Innovative improvements are 
likely to significantly change the 
process/quality performance 


Where appropriate, a proposed 
improvement should be piloted 


Innovative improvements often 
require a phase-in approach 


The effects should be measured 
quantitatively (as opposed to 
qualitatively in OPF), i.e. 
process shift 
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Agenda 


= An Overview of Levels 4 and 5 
= New Behaviors 
= Benefits — Project, Organizational, Customer 
= Making the Business Case 


= Understanding the CMMI Process Areas 
= Organizational Process Performance 
=" Quantitative Project Management 
= Causal Analysis & Resolution 
=" Organizational Innovation & Deployment 


= Strategies for Adoption 
= Lessons Learned 
= Links to Six Sigma 
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Achieving Levels 4 and 5 ts Less Predictable 


= The time it takes to achieve 
Levels 2 and 3 is driven by the 
resources available 
= Learn/implement new project 
practices 
= Create organizational assets - 
policies, processes, training, 
etc. 
= Practices are first performed; 
effectiveness is improved over 
time 


= The time it takes to achieve 
Levels 4 and 5 is driven by the 
ability to stabilize processes 


= Choosing processes that can 
be stabilized 


= Establishing the right metrics 
and methods 


= Collecting enough data 


= Effectiveness is required to 
perform the practices 
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5 Optimizing 


4 Quantitatively 
Managed 


3 Defined 


2 Managed 


1 Performed 


Causal Analysis and Resolution 
Organizational Innovation and Deployment 


Quantitative Project Management 
Organizational Process Performance 


Requirements Development 

Technical Solution 

Product Integration 

Verification 

Validation 

Organizational Process Focus 
Organizational Process Definition 
Organizational Training 

Risk Management 

Integrated Project Management (for IPPD*) 
Integrated Teaming* 

Integrated Supplier Management** 
Decision Analysis and Resolution 
Organizational Environment for Integration* 


Requirements Management 

Project Planning 

Project Monitoring and Control 

Supplier Agreement Management 
Measurement and Analysis 

Process and Product Quality Assurance 
Configuration Management 
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Lessons Learned 


Based on 20 Northrop Grumman CMMI Level 5 organizations 


<—e = Level 3 metrics, measurement processes, and goal setting are generally 
inadequate for Levels 4 and 5 


A a = Better definitions of the measures 
= Lower level metrics of lower level subprocesses 
= Stratifying the data properly 
= When operating at Level 3, it is difficult to predict the measurement 
improvements needed 


= Trying to understand and stabilize the key subprocesses will naturally 
drive you to the right metrics 


= Projects have different quality and process performance needs, and 
should select different subprocesses to quantitatively manage 


= This will also slow adoption, and complicate the organizational baselines 
and models 
= Six Sigma is an enabler for higher maturity 
= Focus on data, measurement systems, process improvement 
= Tying improvements to business goals 
= Tools and methods support the level 4/5 analysis tasks 
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What is Six Sigma? 


= Six Sigma is a management philosophy based on meeting 


7 business objectives by striving for perfection 
A = Adisciplined, data-driven methodology for decision making and 
he process improvement 


= Six Sigma consists of several integrated methods: 
=» Process Management 
= Voice of the Customer 
= Change Management 
= Tools for Measuring Variation and Change 
= Business Metrics 
=» Leading-edge companies are applying Six Sigma to engineering 
work 
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| mportance of Reducing Variation 


*4 = To increase process performance, you have to decrease variation 


oa i f 
4 Too early Too late Too early Too late 
; ag s 
oy Defects Defects . 
y Reduce 
variation 


Delivery Time Delivery Time 


Spread of variation Spread of variation 


too wide compared 
to specifications 


narrow compared to 
specifications 


= Less variation means 
=" Greater predictability in the process 
= Less waste and rework, which lowers costs 
= Products and services that perform better and last longer 


= Happier customers 
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A Typical Six Sigma Project in Engineering 


= Customers express concern that software defects are causing 
—" frequent failures in the field 


= A Six Sigma team is formed to scope the problem, collect data, 
and determine the root cause 


= The team’s analysis of the data determines that poorly 
understood interface requirements account for 90% of the 
problems in the field 


= The interface problems are corrected 


= The organization’s requirements solicitation process is modified 
to ensure future projects do not encounter similar problems 
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DMAIC Process Steps 


Set project goals and objectives 


Narrow range of potential causes and establish 
baseline capability level 


ANALYZE Evaluate data/information for trends, patterns, causal 
relationships and "root causes“ 


IMPROVE Develop, implement and evaluate solutions targeted 
at identified root causes 


CONTROL Make sure problem stays fixed and new methods can 
be further improved over time 
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How Six Sigma Helps Level 4-5 Organizations 


¢ = Six Sigma provides specific 
= 4 mcikode and tools on 
a, ‘ i = Quantitative process 
management of Level 4 
= Causal Analysis and 
Resolution of Level 5 


2 
2 I 


= Six Sigma projects provide a 
mechanism for selecting and 
implementing improvements 

= Addresses Organizational 
Innovation and Deployment 


= Can extend beyond Level 5 


| 
aT 
a 

| 
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Jacobs Technology eeu 


fa JACOBS Lessons Learned from... 


¢ 5 organizations: Jacobs and Government 


¢ About 75 projects 


¢ Over two years 


JACOBS 
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fa JACOBS 


e¢ Technical Director, Jacobs Technology/ITSS 

¢ Candidate SCAMPI Lead Appraiser 

e (lean) Six Sigma Black Belt 

¢ Member of the Steering Committee and Chair of the 
Software Committee, NDIA Systems Engineering 
Division 

¢ Visiting Scientist, Software Engineering Institute 

¢ Project Manager, Software Development 

¢ Systems Engineer, Advanced Research Center 


e Program Manager — Air Force Systems Acquisition and 
Development 


¢ B.S. Aerospace Engineering 
e M.S. Operations Research 


JACOBS 


Problems with “Traditional” Process 


fla JACOBS Improvement Approaches cy 


¢ Monolithic and cumbersome 
— Process deployment cycles of 6 months to 2 years 
— Deployed processes reflect significant changes 


changes at once are difficult to “get right” 
— Latent process defects have a long(er) life 

— Training/implementation/institutionalization costs are high 
— Application to new efforts is delayed 


JACOBS 
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fd JACOBS Objectives 


e By the end of this tutorial, you will: 
— Understand what Lean SPI Is 


— Understand why fast process iterations are critical to 
successful improvement programs 


— Understand how to apply Lean SPI in organizations 
that apply lean or agile development approaches 


— Understand how to apply Lean SPI in “traditional”, 
systemic, plan-based development organizations 


JACOBS 


fla JACOBS But What if We’re Already Lean? 


e Process improvement efforts by agile and lean 
development organizations are often: 
— Undocumented or only partially documented 
— Not under “appropriate” configuration management 


— Difficult (impossible?) to coordinate or learn 
organizationally across development teams 


— Not persistent 


aM alslasvie)as 


Lean SPI 


- Follows lean construct, 
- within “leaned” CMMI framework, JACOBS 
- with Six Sigma process controls. 6 


fl] JACOBS Lean SW Engineering Attributes 


¢ Sharp focus on customer value 

¢ Fast iterations? 4m 

¢ Workflow synchronization 

¢ Early defect removal 

e Rapid learning 

¢ Agile project management 

e Vision point architecture 

¢ Options thinking and “decide as late as possible” 
¢ High degree of concurrency and communication 
¢ Waste elimination 

e¢ Agile engineering support 


Aren’t these good things for process improvement as well? JACOBS 


fla JACOBS Lean SPI — An Outline 


e Focus on the customer 

¢ Kaizen workflow development 

e Process improvement iterations 

e Lean SPI infrastructure 

¢ Agile SPI management 

¢ Process defect removal 

e Summary of CMMI practices and compliance 


Principles can be applied to Systems Engineering PI as well... 
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fa JACOBS Focus on the Customer 


e Lean view of customer value 
— Does the activity or product have DIRECT value in 
the eyes of the customer? 
¢ Configuration management? 
¢ Quality assurance? 
¢ Defect tracking/management? 
e Lean focus on customer needs 
— Quality function deployment 
— Continuous relationship and listening to customer 
— Fast response to customer needs 


JACOBS 
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fa JACOBS Quality Function Deployment 


e Lean/Six Sigma mechanism 


e Proven, high value method for understanding 
customer needs 

e Visual mapping to derived technical 
requirements 

¢ Opportunity to address process quality attributes 
— Speed (of development cycle) 
— Defect profiles 
— Technical domains 
— Deployment of new projects 
— Learning and training constructs 


JACOBS 
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fa JACOBS OFD 


RELATIONSHIPS: INTERACTIONS: 
Strong @ @ Strong Positive 
Moderate © © Mild Positive 


XD XX Strong negative 
& X Mild negative 


| 
: 


Process 
Requirements 


Customer 
Needs 


= 

6 

= 
[<6mo.lifecycle|1 |_| 
No delivered def. oi | 
|EAcompliant [3 | @ | 
[CMMicompliant |5 | | 
100%online [a] | | | 
[Spiral dev. cycles}1 | @ | @| @| O| 


[Technical Difculty(d >| 9] 2| 9] 5] 4 
[mBUrance Rating [iol 9, 91 9] 9 
1-10 


SW Architect 


ole Lean SW Dev. 
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fla JACOBS Lean SPI — An Outline 


e Focus on the customer 

¢ Kaizen workflow development 

e Process improvement iterations 

e Lean SPI infrastructure 

¢ Agile SPI management 

¢« Process defect removal 

¢ Summary of CMMI practices and compliance 


Principles can be applied to Systems Engineering PI as well... 
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fa JACOBS Kaizen Workflow Development 


e What is a Kaizen team? 

¢ Workflow vs. process 

¢ Definition of current state 

e Waste identification 

¢ Definition of ideal state 

¢ Definition of future state 

e Lean projects and tasks 

¢ Configuration management and communications 
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fla JACOBS Kaizen Team 


¢ Kaizen is Japanese for “a change for better” 


¢ Team composition my Process Owners 


— Principal stakeholders 
¢ Those who will use process 
¢ Those who will be directly affected by process 


e Team charter 
— Chartered by senior management 
— Authority to “do what is needed” 
— Own the process..... forever 
— Resources to do job 


e Trained and led/facilitated 


: JACOBS 
Kaizen Workflow Development ia 


fla JACOBS Workflow vs. Process 


Workflow: 
— operational aspect of a work procedure 
— describes how tasks are structured 
— who performs tasks 
— what their relative order is 
— how they are synchronized 
— how information flows to support the tasks 
— how tasks are being tracked 
— considers "throughput" as a distinct measure 


e Process: 


— sequence in which properties or attributes of a system or object 
are changed 

— sequence of operations and involved events, taking up time, 
space, expertise or other resources, leading to the production of 
some outcome 


: JACOBS 
Kaizen Workflow Development ie 


fl] JACOBS Workflow vs. Process (CCB Action) 


Process: 


Configuration 
Control Deploy Product 


Submit Prepare 
to for 


Develop 


ECP CCB Approval 


Deploy 


Deployment Pyatelte 


6 i 1 | 10 1 2 3 3] 6Qncors 


fla JACOBS Definition of the “current state” 


¢ Workflow of how it’s being done now 
e Don’t worry about warts or things that are broken 
¢ Don’t worry about who’s “at fault” 


¢ Starting from “the right hand side” often useful 
— Start with end product or task and work backword 
— “How did we get this?” 
e Discovery: 
— May be the first time the workflow has been defined 
— oe the first time workflow participants Know who really does 
wha 
e Fully define workflow 
— Who does each task 
— Resources and information used 
— Products produced 
— Performance measures 
— Time to complete (measured or estimated) 
— Takt time (calculated, not measured) 
¢ German for “rhythm” or “beat” 
¢ Takt time = Net available time in a day / customer demand per day 


Kaizen Workflow Development 


fla JACOBS Discussion of Current State 


Will often represent a completion time that is 
(much) longer than the takt time 


Development (and discussion) will reveal some 
intuitively obvious opportunities for improvement 


e Discussion of takt time induces focus on 
customer needs 


e Discussion of information flow Is typically 
revealing 


e Early revelations of needs for synchronization 


How to you identify “waste” in the workflow? 


JACOBS 
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Kaizen Workflow Development 


fa JACOBS Identification of Waste (muda) 


e Anything that is not of value to the customer CM and OA 
e¢ Partially done work 

e Extra processes 

e Extra features BValeameyatsssmanltelaia elsmansctciciarem olU imc marcverctsct-la ats 


¢ Task switching mM alksmlalelanatclicelamivilimelsmUtssvomlamuarsmelsiilalitiele 
¢ Waiting of the Ideal State 


e Motion 
¢ Defects 
e Management activities Lean oversight and approval 


acum feces frome 
Develop Sul CCB Piel pelie Deploy 
Product 


-—> to t——> t—>| for t——>| 
EGP Approval Deployment 


JACOBS 


6 1 1 10 1 2 3 3 60 


fa JACOBS Kaizen Workflow Development 


¢ What Is a Kaizen team? 
e Workflow vs. process 
¢ Focus on the customer 


¢ Definition of current state 
@ Waste identification + Process improvement iterations 


* Lean SPI infrastructure 


| Ea vacoss Lean SPI — An Outline 


¢ Definition of ideal state Semen 
oo. * Summary of CMMI practices and compliance 
bas D efi Nn tl O Nn of futu re state aa Principles can be applied to Systems Engineering PI as well... 


e Lean projects and tasks 
¢ Configuration management and communications 


JACOBS 
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fla JACOBS Definition of Ideal State 


e Conceptually remove all constraints: 


Assume authority to “make it the way it should be” 
No tool or technology constraints 
Assumes processes are fully developed and stable 


Assumes information systems are in place and operate with no time 
delay 


e Often requires (re)negotiation of who does what 
e All identified waste should be eliminated 


Anything that is not of value to the customer 
Partially done work 
Extra processes 


Extra features an , — 
Task switching (am (am WAVdal Col aie) aTosswas'ce)6} (0M elomeliai(e1el| mie) an e)celele-lanlanl< esi 


Waiting 
Motion 


Defects O 


Management activities 


"Adal (ovale) atoxomriy(e)0} (0m elomeliii(eveliance) maat-lat-\e(<) aowa 


e Takt time should be met 
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fl] JACOBS Definition of the (15‘) Future State 


e Real-world constraints 
— Some things cannot be assumed away 
— Some tools and mechanisms take time to develop 
— Somewhere between current and ideal states 
¢ Aim for real improvement in the near future 
— Improve key performance measures 
— Pay attention to takt time 
— Accept waste with great “regret” 
e Fully define future state 
¢ Re-estimate performance measures 
¢ Get absolute commitment of Kaizen team members 
¢ Define tasks, activities, and “projects” needed to put in 
place 
¢ Kaizen TEAM briefs management for approval 
JACOBS 
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fa JACOBS Kaizen Workflow Development 


e What is a Kaizen team? 
e Workflow vs. process 


| fa JACOBS Lean SPI — An Outline 


e Definition of current state eee «tA, ey 


@ Waste identification + Process improvement iterations 


*« Lean SPI infrastructure 


¢ Definition of ideal state Ie preaieestver nna 
ae | * Summary of CMMI practices and compliance 
e D ef Nn tl O Nn of futu re state aa Principles can be applied to Systems Engineering PI as well... 


e Lean projects and tasks 
¢ Configuration management and communications 


JACOBS 
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fa JACOBS Lean Projects and Tasks 


e Things to accomplish to achieve Future State 
¢ Workflows are transformed into processes 


e Lean Projects 
— Duration greater than perhaps a few weeks 
— Require significant resources/coordination 
— Complex in nature 
¢ Tool instantiation for the workforce 
¢ Definition of kanban visualization metrics 
e Lean tasks 
— Simple “to-dos” 
— Track and complete 
e All projects and tasks tracked to completion 


JACOBS 
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. Configuration Management and 
JACOBS Communications 


e Kaizen events do not occur In a vacuum 


¢ Kaizen events: 
— Depend on the output products from other Kaizens 
— Produce products that will be used by other Kaizens 
e Process configuration management is key 


— Kaizen teams provided authority to work on workflows and 
processes 


— Respect for CM boundaries crucial 

¢ Communication among Kaizen teams is key: 
— Clarify timing of Kaizen events 
— Understand and manage input/output relationships 
— Respect Kaizen boundaries and “ownership” 


JACOBS 
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fla JACOBS Lean SPI — An Outline 


Focus on the customer 

Kaizen workflow development 

Process improvement iterations 

Lean SPI infrastructure 

Agile SPI management 

Process defect removal 

Summary of CMMI practices and compliance 
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fla JACOBS Process Improvement Iterations 


¢ What IS a “process improvement iteration”? 
— A deployment of a standard organizational or define process or 
Subprocess 
— Possible timelines (use flipchart) 
e 3 years 
e 2 years 
e 1 year 
¢ 6 months 
¢ 3 months 
¢ 1 month 
— Includes: 
¢ tools/mechanisms 
° processes/subprocesses 
e training 
¢ deployment instructions 
e Aren't frequent process iterations a really bad idea? 
— Don't project personnel dislike constant change? 
— Aren’t stable processes a good idea? JACOBS 
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Lean SPI: SYNCHRONIZE process improvements 
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fl] JACOBS “Stable” Processes Peas 


e The “layperson’s” view 
— Processes should not change, except once in a great while, and 
we’re not so sure about that” 
— “If processes change all the time, how do | know what to do?” 
— “Doesn't constant change cause a lot of inefficiency, and indicate 
we don’t know what we’re doing?” 
¢ The Six Sigma (and CMMI) view: 
— Process stability = removal of special causes of variation 
e (Causes that are external to and beyond the scope of the process) 
— Stable process are controlled, but there are degrees of control, 
e.g.: 
e Variance in defect rates 
e Variance in completion times 
¢ The lean view: 


— Continuous, immediate, piece-meal improvement, by the 
process owners, is key to success 


JACOBS 
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fd JACOBS Dealing with Stability 


e Education 
— From top to bottom in the organization 
— Continuous 
— Sometimes even works 
e Change the expectation: 
— Off-load onerous “training” requirements for @ process iteration 
¢ Online, JIT training or point-of-use instructions 
e Shared knowledge mechanisms 
— Simple, absolute, unequivocal clarity on who/when will use new 
process/ subprocess 
— Successes and “lessons learned” in small, quick, improvements 
e Lean improvements, so buy-in is not a problem 
¢ Implement quickly 
e Institutionalize quickly 
e Advertise the successes 


e Let the project or iteration complete the cycle with the process they 
started with 


— Exceptions MAY occur JACOBS 


fla JACOBS Synchronization of PI Iterations (1 of 2) 
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= Possible Issue: 

= Synchronization of 
> Lean SPI Infrastructure 
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fla JACOBS Synchronization of PI Iterations (2 of 2) 


Project B Project A 


Project AC 


OSP 


fla JACOBS Lean SPI — An Outline 


Focus on the customer 

Kaizen workflow development 

Process improvement iterations 

Lean SPI infrastructure 

Agile SPI management 

Process defect removal 

Summary of CMMI practices and compliance 
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fla JACOBS Lean SPI Infrastructure 


e Why do! care about an infrastructure? 
e¢ (Organizational) process architecture 
e Agile process support 

e Concurrency and communications 

e Organizational Learning 


JACOBS 
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fla JACOBS Why Do | Care? (about an infrastructure) 


¢ | shouldn't care if... 
— | want to re-accomplish my rapid improvements 


— | want Kaizen teams to operate without the benefit of 
knowledge of other teams’ work 


— | don’t worry too much about confounding other parts 
of the process when | fix or improve something 


— | like to do things the hard way.... 
¢ | should care if... 


— | want improvements to “click in” to place 
— | want improvements to persist 
— | don’t want to confuse the workforce 


Ye lamlalicetsiigu (eli |comenmrel0|i(omcjul 0) ele) am e)celectoorom-lalemenclearclalsiace JACOBS 
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fla JACOBS Organizational Process Architecture 


e Processes 
— Processes 
— Subprocesses 
— Relationships 


e Map to technologies 


— Requirements definition and management The Process 
Design Architecture 


Code construction 
— Test 
— Metrics 
— Oversight...... 
e Map to mechanisms 


¢ Rapid improvement efforts 
— Use to charge Kaizen teams and focus their work 
— Use to migrate input/output of Kaizen teams to the right places 
— Use to conduct process configuration audits 


JACOBS 
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fla JACOBS Agile Process Support (1 of 2) 


¢ What is agile process support? 

“Pulled” by kanban visualization 

— Fully prepared to act when “pulled” 
e “Pull the trigger” 


— Gem atom (ckcTeKswrelalemcy (creel tcsom eral 
SPI efforts (e.g. Kaizens)? 
- What BmmAAVaTo mee) i\/atcm talcum €)€0letctsssiars 


— Kaizen events 

— Improvement task or activity support 

— Strategy sessions 

— Process configuration management actions 
— Process QA actions 

— Process verification and validation actions 


or take 
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fa JACOBS = Agile Process Support ¢ of 2) 


e Kaizen events 
— Instantiate (respond) quickly to the problem 
— Conduct in an orderly, well-prepared manner 
e (Kaizen process, mechanisms, tools, are in place) 
— Build out-brief quickly (hours) and get full consensus 
— Provide output products quickly (a day or less) 


¢ Process configuration management actions 
— Kanban pulled trigger to accomplish configuration audit 
— Immediate reporting of CM audit results to stakeholders 
— Mechanisms and oversight to close CM defects in hours or days 


¢« Process QA actions 
— Kanban pull process and work product audits] 
— Audit quickly, targeting planned audit items and using criteria 
— Report audit results on the spot to all affected stakeholders & mgt 
— Provide mechanisms and oversight to close defects in hours or 
days 
JACOBS 
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fla JACOBS Concurrency and Communications 


¢ Lean SPI invokes a higher standard 


e Relevant stakeholders are aware of events and activities 
(and their roles) in minutes or hours 


¢ Communications are pushed whenever possible 
— Supports rapid decision making 
— Enables immediate instantiation of follow-on activities 
e Examples: 
— Electronic status boards 
— Email trigger for desk top notification 
— Cellular notification 
e Kaizen events mandate buy-in (and support) of team 
members for Kaizen decision and products 
JACOBS 
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F¥JACOBS = Organizational Learning 


e Classroom learning is an anti-pattern to lean learning 
constructs 
— Relatively long preparation times 
— Relatively inefficient knowledge transfer/assimilation 
¢ Difficult to get attention and engagement 
¢ Cost inefficient 
— Relatively disruptive to the flow of work 
e Let’s stop and learn(!!) 
e What other (faster and more continuous) ways are there 
to learn? 
— Learn during Kaizen events 
— Team members investigation and knowledge-sharing 
— Knowledge base development/ knowledge mat. 
Lessons learned sharing and implementation in process 
Immediate sharing of quality audit results across organization 
Immediate sharing of CM audit results across organization 


JACOBS 
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fa JACOBS 
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fla JACOBS Lean SPI Infrastructure 


e Why do! care about an infrastructure? 
e Process archite Gijseg@ : , 

BM alssmexe) an) e)ioioxswe | oven kssyle) a 
e Agile process SUBSE WES leis 
e Concurrency and communications 
¢ Organizational Learning | iiacoss Lean spi—An Outline 


« Focus on the customer 


* Kaizen workflow development 


‘rocess improvement iterations 


| « Lean SPI infrastructure 
- | [2 Agile SPl management 


« Process defect removal 


« Summary of CMMI practices and compliance 


JACOBS 


fla JACOBS Lean/Agile Project Management 


¢ Seeing waste How would 

e¢ Value stream mapping these apply to 
e Feedback SPI management? 
e Iteration leadership/management 

e Options thinking 

e Last responsible moment decision making 

¢ Pull/Kanban systems and measurements 

¢ Cost of delay awareness 

¢ Self determination/team empowerment 

¢ Motivation and leadership 

¢ Technical expertise 

e¢ Refactoring (design against more stable architecture) 


JACOBS 
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fa JACOBS Agile SP! Management 


¢ Remain sharply focused on the needs of the customer(s) 
— Reassess customer needs continuously 
— May (will probably) change over time 
¢ Sometimes quickly 
— Understanding of customer needs will get better 
e Ensure development “heroes” are in lean leadership 
positions 
e Listen to the software development workforce 
— Kaizen improvements should be initiated within the workforce 
— Support workforce resolution of problems and implementation of 
improvements (agile support) 
e Improve and continuously tune the process architecture 


e Make “last responsible moment” investment decisions 
— In Kaizens or other improvement efforts 
— In technologies 


JACOBS 
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fa JACOBS Lean SPI — An Outline 


Focus on the customer 

Kaizen workflow development 

Process improvement iterations 

Lean SPI infrastructure 

Agile SPI management 

Process defect removal 

Summary of CMMI practices and compliance 
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fa JACOBS Process Defect Removal 


e Lean focuses sharply on identification and removal of defects at the 
earliest possible moment 
e¢ Fast, “immediate” removal of defects is the goal 
— Ver/val of process elements prior to deployment 
— QA and CM audits yield defects 
— Process simulation can reveal defects 
e Rapid, continuous improvement provides continuous opportunities 
to remove defects 
e Six Sigma defect elimination ( 
— DMAIC cycle 
— Process control (reduction in variation = fewer defects) 
e Poka Yoke (mistake proof) — mechanisms to eliminate the possibility 
of defects 
— Well constructed checklists 
— Coding rules 
e Muda - waste elmination 
JACOBS 


Ea JACOBS CMMI Summary 
e Lean SPI processes are Managed processes 
— Process relationships are tracked 
— Voice of the customer and stakeholders are key 
e Lean SPI processes under six sigma control are 
quantitatively managed processes 
e CMMI-DEV process areas most germane to this 


discussion: 
— OPF 

— OPD 

— OT 

— OID 


JACOBS 


fl] JACOBS Lean SPI view of OPF 


Purpose 

The purpose of Organizational Process Focus (OPF) is to plan, 
implement, and deploy organizational process improvements based on 
a thorough understanding of the current strengths and weaknesses of 
the organization’s processes and process assets. 


SP 1.3 Identify the Organization's Process Improvements 
Identify improvements to the organization's processes and process 


ASSErs: Improvements identified primarily by customers and process owners 


SP 2.1 Establish Process Action Plans 
Establish and maintain process action plans to address improvements 
to the organization's processes and process assets. 


A: IyA-1a core lia merarc Vans) acy) e)t-lalcw- ale mip @cy(elait-m e)ge)(-ceam e)t- lays 


SP 3.4 Incorporate Process-Related Experiences into the Organizational 
Process Assets 

Incorporate process-related work products, measures, and improvement 
information derived from planning and performing the process into the 
organizational process assets. 


Near real time, thru metrics, Kaizen events, lessons learned and 


fl] JACOBS Lean SPI view of OPD 


Purpose 

The purpose of Organizational Process Definition (OPD) is to establish and 
maintain a usable set of organizational process assets and work environment 
standards. 


SP 1.5 Establish the Organization’s Process Asset Library 
Establish and maintain the organization's process asset library. 


The PAL is responsive and supports rapid improvement 


SP 2.1 Establish Empowerment Mechanisms 
Establish and maintain enpowerment mechanisms to enable timely 
decision making. 


Kaizen teams, kanban visualization, poka yoke, muda 
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fla JACOBS Lean SPI view of OT 


Purpose 


The purpose of Organizational Training (OT) is to develop the skills and 


knowledge of people so they can perform their roles effectively and 
efficiently. 


yNamlavce)eaatstilissmsi(siaalsiale 


Certain skills may be effectively and efficiently imparted through 

vehicles other than in-class training experiences (e.g., informal mentoring). 
Other skills require more formalized training vehicles, such as ina 
classroom, by Web-based training, through guided self-study, or 

via a formalized on-the-job training program. The formal or informal training 
vehicles employed for each situation should be based on an assessment of 
the need for training and the performance gap to be addressed. The term 


“training” used throughout this process area is used broadly to include all of 
these learning options. 


JACOBS 
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fl] JACOBS Lean SPI view of OID 


Purpose 

The purpose of Organizational Innovation and Deployment (OID) is to select 
and deploy incremental and innovative improvements that measurably improve 
the organization’s processes and technologies. The improvements support the 
organization’s quality and process performance objectives as derived from the 
organization’s business objectives. 


SP 1.1 Collect and Analyze Improvement Proposals 
Collect and analyze process- and technology-improvement 


r Is. ; i 
el aia “Collect” -— some over time, most perhaps near-real-time 


SP 1.2 Identify and Analyze Innovations 
Identify and analyze innovative improvements that could 
increase the organization’s quality and process performance. 


But keep it Lean — focused on customer value and fast 


SP 1.3 Pilot Improvements 
Pilot process and technology improvements to select which 
ones to implement. 
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fla JACOBS Lean SPI — An Outline 


Focus on the customer 

Kaizen workflow development 
Process | WAV owre lace (0) ated 
Lean SPI 
Agile SPI 


Process (¢ Questions? 


Summary of CMMI practices and compliance 


fla JACOBS Contact Information 


Jeffrey L. Dutton 

6703 Odyssey Drive, Suite 304 
Huntsville, AL 35803 
256.971.5527 


jeff.dutton@jacobs.com 


Please see our web site for additional information and references. 
http://www.jacobstechnology.com/cmmi/welcome.htm 
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Contact Information 


Introduction 


@ In order for organizations to change and improve the way they 
conduct business they have to be ready to accept change 


@ Before attempting process improvement, or other initiatives such 
as integrating systems of systems, organizations need to 
determine if they are ready for change 


@ When new initiatives are attempted in organizations that are not 
prepared to accept change the efforts are doomed to fail 


@ This is demonstrated by many recent incidences of: 
> Failures while developing large complex systems 

Lack of information sharing among agencies 

Inability to integrate large weapons system 

Etc., etc., etc. 
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Introduction (cont.) 


® 


¢ 


Many commercial and Government organizations and agencies 
are stove piped and dysfunctional 


Organizations may be in chaos and disarray and may not 
function as cohesive, seamless units 


Problems may include the inability of organizations to 
communicate, take responsibility and ownership, synchronize 
and synergize, communicate and share information 


Many times organizations reorganize when they encounter 
issues and problems 
> This really does not fix the problems 
« They remain and become the problems of the new organizational 
structure and of new management 
« They may be further hidden where they become graver and are more 
difficult to identify 


Introduction (concl.) 


@ When integrating stove piped legacy Systems-of-systemMs we tend 
to concentrate on technical issues associated with: 
> Interoperability 
> Interfaces 
@ We are not recognizing that major problems are associated with 
organizations that own these stove piped systems that are: 
> Themselves stove piped 
> Many times dysfunctional 
@ What needs to happen first is that organizations need to: 
> Get rid of this stove piping 
> Fix their dysfunctional states 
@ Only then can legacy systems be integrated into systems-of-systems 


Where Are We? 
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Characteristics of a dysfunctional Organization 
se pees) Hidden 
fighting snendae 
Hero Rewarding . 
mentality inappropriate DUreaucracies 
behavior gone amuck 
Time to 
Closed Start over 
cultures 


Don’t talk 


None of my 
business 


Attending 
meetings but not 
committing to anything 


Sabotage the 
meeting 


It's mine 
not yours 


Hostile 
policies 


Does not 
Lack of belong to me 


synergism 


Failure to 
communicate 


No 
information 
sharing 


Canceled 
project/ 
initiatives 


reorganize 


What are the Characteristics of a Functional 
Organization? 


Stove Piped Organizations 


Cooperation/Communications/Information Sharing? 


NO 


Organizations 


What we have here is a failure to communicate. / can’t believe | said that! 


Attempting to do PI in a Stove-Piped 
Dysfunctional Organization 


ipsa in: Band-Aiding the problems 


Band-Aids Fall Off 


And in Spinning Your Wheels 


And failure! 
And failure! 


And failure! 


And failure! 


Where Are We? 
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A Solution 


@ Get to the root causes of the problem(s), fix the 
organization - Conduct an Organizational Analysis 
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What is Organizational Analysis? 


@ An organizational analysis is an assessment of an organization to 
determine the current “As-Is” state of the organization, to propose a 
“To-Be” state, and to develop a plan and schedule to achieve the 
desired state 

@ The intent is to uncover and correct root causes of problems and 
break down barriers which prevent organizations from functioning as 
cohesive and seamless units. This results in an organization where: 

> Information is shared 

Communications are wide spread 

Roles and responsibilities are assigned 

Ownership is accepted 

Accountability is exercised 

Synergism is the rule 

Re-work is minimized 

Bureaucracies are non-threatening 

Cost and schedules are adhered to 

Disarray and chaos are non-existent 

Fire-fighting is minimized 
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Configuration of Organizations 


Levels of Organizational Analysis 


Organizational Analyses Conducted 


3rd 
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What Elements are investigated during an 
Organizational Analysis? 


Mission 


& Ne @ Activities 

Vision @ Accountability 

@ Strategic Goals @ Ethics 

@ Charters # Communications 
@ Organizational Structures @ Information Sharing 
@ Contractors @ Facilities 

@ Suppliers @ Resources 

@ Customers @ Tools 

@ Stakeholders @ Personnel 

@ Products @ Skills 

@ Services @ Training 

@ Roles @ Policies 

@ Responsibilities @ Processes 
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What Elements are investigated during an 


Organizational Analysis? (concl.) 


Mission, 
Vision, 
Charter 


Customers, 
Products, 
Services 


Organizational 
Structure & 
Facilities 


Policies & 
Processes 


Strategic 
Goals & 
Activities 


Contractors, 
Products, 
Services 


Roles & 
Responsibilities 


Procedures 
& Tools 


Staff & 
Training 
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Roles & 
Responsibilities 


Relationships 


Policies & 
Procedures 
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Some Organizational Elements Defined 


@ MISSION - The mission of an organization relates top-level 
business objectives that an organization is chartered to 
accomplish and support. The mission is usually reflected in a 
succinct and precise statement that the organization constructs, 
endorses, and follows. 


@ VISION — The vision is where the organization desires to advance 
itself in the future in relation to expanding or changing its mission. 


@ STRATEGIC GOALS -— Strategic goals are high-level goals that 
the organization has to achieve in order to meet its mission and 
vision. 

@ CONTRACTORS -— Contractors provide products and/or services 
to the organization. 


@ CUSTOMERS - The organization’s customers are those that 
receive products and/or services from the organization. 
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Some Organizational Elements Defined (cont.) 


@ KEY STAKEHOLDERS - Key individuals or groups that have a 
vested interest in the activities, products and services of your 
organization. 

@ ACTIVITIES — Actions that need to be executed and accomplished in 
support of developing, acquiring, and maintaining products and 
Services. 

@ ROLES -— The function and position of individuals and their expected 
behavior within the organization and with other stakeholders. 

@ RESPONSIBILITIES — Things for which individuals within the 
organization are responsible such as duty, obligation, and burden. 

@ STAFF — The individuals within the organization that need to have 
the appropriate skills and experience to satisfy their assigned roles 
and responsibilities. 

@ TRAINING — The training required to ensure that the staff obtains the 
required skills to satisfy their roles and responsibilities. 
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Some Organizational Elements Defined (cont.) 


@ TECHNOLOGY INSERTION — Keeping up with technology 
innovations and the insertion of advanced technologies into the work 
environment and products. 


@ ETHICS — The rules and standards that govern the conduct of 
members of an organization and the general nature of morals and of 
the specific moral choices made by individuals in relationship to others. 

@ COMMUNICATIONS - The ability of individuals within the organization 
to communicate with each other, with interfacing organizations, and 
with other stakeholders. Open and clear lines of communications are 
necessary for individuals to adequately perform their roles and 
responsibilities. 

@ INFORMATION SHARING - The ability of organizations to share 
valuable information within the organization, with stakeholders, and 
with interfacing organizations. 
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Some Organizational Elements Defined (cont.) 


@ PROCESS - The key activities that individuals conduct day-to-day to 
satisfy their roles and responsibilities and to produce or acquire 
products and services. Processes need to be documented, consistent 
and repeatable for both technical and management activities. 
Examples are processes for requirements management, configuration 
management, quality assurance, project planning, and risk 
management. 


@ PROCEDURES - Instructions that individuals follow in the execution of 
a specific process including the steps necessary to accomplish the 
activities within the process. 

@ STANDARDS -— DoD or industry regulations that are required to be 
addressed when developing or acquiring products and services. 
Examples are documentation standards, safety standards, security 
standards, performance standards, and quality standards. 
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Some Organizational Elements Defined (concl.) 


@ FACILITIES — Physical building and rooms where the organization is 
housed including furniture and the proximity of individuals to each 
other. 

@ RESOURCES - Items required to ensure that staff can accomplish 
their roles and responsibilities, to develop and receive products and 
services, and to execute assigned activities. Examples of resources 
are budgets, funding, equipment, tools, and administrative support. 
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Summary 

Reference 

Questions/Answers/Discussion/Comments 

Contact Information 
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The Big Picture 


Where are you now? 
Where do you want to get to? 


What’s the best way to get there? 


How is an Organizational Analysis Conducted? 


Get agreement with the organizations to do an organizational analysis 
Develop a plan for the analysis to include: 
> Objective of analysis 
> Elements to review 
> Schedule for analysis 
> Identification of participants 
Request documentation/artifacts on elements for review 
Construct 
> Questionnaire and/or 
> Interview questions 
On organizational elements 
@ Interview and/or send out questionnaire to: 
> Executives 
> Managements and leads 
> Key staff members 
> Key stakeholders 
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How is an Organizational Analysis Conducted? 
(concl.) 


Questions need to include: 
> What is the current state? 
> What do you think the future state should be? 
«* What do you want to be when you grow up? 
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Organizational Analysis Questionnaire 


@ An example of typical questions asked about Strategic Goals: 
> Does the organization have strategic goals? 
> Do you know what the strategic goals of the organization are? 
> Are the strategic goals well published? 
> Does staff receive exposure to the strategic goals? 
> If there are no strategic goals what do you think they should be? 
> Please provide additional comments 
* Will not be attributed to individuals 
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Results can be Depicted in an Executive 
Dashboard 


Mission | 
Vision | 
Strategic Goals | 
ae ae 


Policies 


Processes | ON | | 
Procedures | _ BLUE is best 
a GREEN 


YELLOW 


ORANGE 


RED is worst 


Next the organizations prioritizes the elements in order of importance 
(As-Is state analyses and prioritization shown later in example) 33 


Next 


@ Resolve misunderstandings/issues/conflicts in the results. May 
require: 
> Brain-storming sessions 
> Additional objective non-attribution investigations 
* Questionnaires 
* Interviews 
@ The respondents agree on the results 
@ Evaluate Priorities vs. Executive Dashboard results 
@ Develop a “To-Be” strategy and plan 


> Based on the evaluation of Priorities/Executive Dashboard/Comments 
Received 


« Near term plan 

« Mid term plan 

“ Long term plan 
@ Execute plan 


Use of Priorities vs. Executive Dashboard 


@ Where a high priority item rates low in the dashboard 
> May be selected for near term plan 


@ Where a mid priority item rates medium in the dashboard 
> May be selected for mid term plan 


@ Where a low priority item rates high in dashboard 
> May be selected for long term plan 
> May not be selected for plan 


BLUE is best 


GREEN 
YELLOW 


ORANGE 


RED is worst 
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Comments Provided 


@ Can be used 

To develop an understanding of organizational issues 

As stepping stones for further discussions and ideas gathering 
To harvest ideas for “To-Be” state 

To support development of “To-Be” plan 


V V V WV 
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Example 1 


@ At aworking session with a new DoD client, while starting to 
implement CMMI using a FASTrack? approach it was mentioned 
that the Director was planning to reorganize the organization 

@ | mentioned that too often organizations reorganize for the wrong 
reasons especially when things are not going right 

> They think that reorganizing solves problems 
> What really happens is that they are masking the problems and not 
getting to their root causes 

@ | mentioned that | had conducted Organizational Analysis (OA) on 
organizations to uncover root causes of problems, fix them, and 
prevent them from recurring 
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Example 1 


@ | briefly explained what an Organizational Analysis (OA) is 


> | mentioned that an OA may discover that a reorganization may not be 
necessary 


> But if reorganization is necessary, they now have a valid reason anda 
road map for this reorganization 


@ They got excited and suggested that | present my approach to the 
Director 


@ A meeting was arranged and | pitched my OA approach 
@ At first she hesitated 

> Was initially apprehensive about what she might learn 
@ Later she was willing to accept the consequence 

> Asked me to conduct the OA 
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Very Important 


When explaining OA you need to state that you do not mean to 
Imply that: 
@ Their organization is stove piped and/or dysfunctional, BUT: 
> Most suffer from some of the symptoms presented 
> Any one or any number of these may be impediments to change 


Example 1 


Questionnaire Developed 


@ Only questionnaire used 

@ No interviews conducted 

@ This method is faster and less intrusive than interviewing 
> But does not yield as much information 

@ The following 20 elements were in the questionnaire: 


> Mission > Training 

> Vision > Technology Insertion 
> Strategic Goals > Ethics 

> Contractors > Communications 

> Customers > Information Sharing 
> Key Stakeholders > Process 

> Activities > Procedures 

> Roles > Standards 

> Responsibilities > Facilities 

> Staff > Resources 


Al 
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Questionnaire Sent Out 


ORGANIZATIONAL ANALYSIS QUESTIONNAIRE 
An organizational analysis is an assessment on an organization to determine the current “As-Is” 
state of the organization, to propose a “To-Be” state and to develop a plan and schedule to 


achieve the desired state. The intent is to uncover and correct root causes of problems that 
may exist which prevent an organization from functioning as a cohesive and seamless unit 
where: 

@ Information is shared 

@ Communications are wide spread 

@ Roles and responsibilities are assigned 

@ Ownership is accepted 

@ Accountability is exercised 

@ Synergism is the rule 

@ Re-work in minimized 

 Bureaucracies are non-threatening 

@ Cost and schedules are adhered to 

@ Disarray and chaos are non-existent 

fire-fighting is minimizes 
Enter “YES”, “NO”, “DNK” if you do not know, and “NA” if not applicable. If the answer is “No” go 
to the next logical question. You are encouraged to insert comments that further describe your 
knowledge of the area that the question refers to. These comments will support the 
establishment, enhancement and understanding of the specific area and will support the 
definition of the “To-Be” state. The final report will consolidate responses and will not attribute 
responses to individuals; feel free to express your true feelings. The only person that will see 
the raw information is independent of the organization. A? 


Name of Individual: Date: 


Example 1 


Questionnaire Responses (Sample as Returned) 


@ Following are samples of returned questionnaires 


@ Each individual returned a complete set for all elements in the 
survey 
> All elements presented earlier were used 


# The samples here are a few from only one individual 
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Example 1 


Questionnaire Responses (Sample as Returned) (cont.) 


QUESTIONS COMMENTS 


MISSION - The mission of an organization relates top-level business objectives that an organization is charted to accomplish and 
support. The Mission is usually reflected in a succinct and precise statement that the organization constructs, endorsed and follows. 


VISION -— The vision is where the organization desires to advance itself in the future in relation of expanding or changing its mission. 


Do you know what the vision of the organization is? 
Does staff receive exposure to the vision? 


If there is no vision what do you think it should be? 


The vision is readily available, but many employees are 
not aware of j 

STRATEGIC GOALS - Strategic goals are high-level goals that the organization has to achieve in order to meet its mission and 
vision. 


Does the organization have strategic goals? 
Do you know what the strategic goals of the organization are? 
Are the strategic goals well published? Strategic Goals: Establish open communication with the 
D ff ; h ; 15? users (Test Teams, System Developers, etc.). Explore all 
DES Sta ECE Ve EADOSUIE We Sueeeee areas of funding. Maximize testing and training where 


If there are no strategic goals what do you think they should be? possible. 44 


Example 1 


Questionnaire Responses (Sample as Returned) (cont.) 


QUESTIONS 


and with other stakeholders. 


Are there descriptions of the various roles within the 
organization? 


Are these documented and can you briefly describe the ones t 
hat apply to you? 


Does your manager or supervisor review them with you 
initially and as needed? 


Do you review them with your subordinate initially and as 
needed? 


ROLES -— The function and position and their characteristics of individuals and their expected behavior within the organization 


COMMENTS 


There are job descriptions, but nothing on how to 
support or manage stakeholders. Nor have I seen 
anything on the functions of individuals or expected 
behavior to conduct within the organization. 


N 
N 


RESPONSIBILITIES — Relates to things for which individuals within the organization are responsible for such as duty, 


obligation, and burden. 


Are there descriptions of the various responsibilities within 
the organization? 


Are these documented and can you briefly describe the ones t 
hat apply to you? 

Does your manager or supervisor review them with you 
initially and as needed? 


Do you review them with your subordinate initially and as 
needed? 


N 
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Example 1 


Questionnaire Responses (Sample as Returned) (concl.) 


QUESTIONS 


COMMENTS 


COMMUNICATIONS - The ability of individuals within the organization to communicate with each other, with interfacing 
organizations, and with other stakeholders. Open and clear lines of communication are necessary for individuals 
adequately perform their roles and responsibilities. 


Does the organization have open door policy with DNK 
management? 


How useful is it? 


Does the organization identify key points-of-contact (POC) 
for communicating important information? 


Are POCs published? 


Are informational and status meetings held with the staff and Y 
how often? 


Are informational status reports generated and how often? 
Are teambuilding sessions held and how often? 
Do you find them useful? 


Can you please make any recommendations to ensure that individuals 
have open and clear lines of communications with management, each 
other and all stakeholders? 


Walk around and inform, short weekly papers, or post 
notes. 
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Example 1 


Questionnaire Results 


@ Nine responses were received (including one from director) 
@ The following compiled results reflect the “As-Is” state of the 


organization: 
> Executive Dashboard (Red, Orange, Yellow, Green, Blue) 


> Based on the responses to questions 


47 


Example 1 


Questionnaire Compiled Results (sample) 


RESPONSES 12345 
MISSION 
Do you know what the mission of the organization is? 
Is the mission well published? 
Does staff receive exposure to the mission? 
Totals 


1 11 
1 1 


ORR P 
NPR 


VISION 

Does the organization have a vision? 

Is the vision well published? 

Do you know what the vision of the organization is? 
Does staff receive exposure to the vision? 

Totals 


RRPRPRPPR 


0000 


STRATEGIC GOALS 

Does the organization have strategic goals? 

Do you know what the strategic goals of the organization are? 
Are the strategic goals well published? 

Does staff receive exposure to the strategic goals? 

Totals 


RRPRPRP 


0000 


ORR P 


i<e) 


ORR P 


ee 
EP wR WW 


3 


COMNN NM 
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Consolidated Results- Executive Dashboard 


py 
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Individuals Responding— 1 


Mission 

Vision 

Strategic Goals 
Contractors 
Customers 

Key Stakeholders 
Activities 

Roles 
Responsibilities 
Staff 

Training 
Technology Insertion 
Ethics 
Communications 
Information Sharing 
Process 
Procedures 
Standards 

Facilities 
Resources 


ow 
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27 
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Example 1 


Key to Executive Dashboard 


py - Possible yes's per item TY - Total number of yes's for all responses 


PY=py*9 - Possible yes's for allresponses %Y=TY/PY - Percent of yes's 


Color=%Y 
RED=<39% ORANGE =40%-69% YELLOW=70%-84% 


GREEN=85%-94% BLUE=>95% 
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Example 1 


Comments Received 


@ The following are samples of comments received 
@ More that 250 comments were received 
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Example 1 
Comments Received (cont.) 
@ Mission 


> 
> 
> 


The mission statement too broad. 
The mission statement lacks focus and direction. 


| feel that very few individuals internal and external understand the 
mission and several internal are not working to accomplish it. 


Don't recall ever seeing a mission statement. 


Mission ts published in official document available on the Intranet and 
the mission is used in most briefings. On the other hand, | know from 
recent discussions that some don't know the mission clearly. 
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Comments Received (cont.) 


@ Roles 


> Chief of Ops Division provides technology support to all the local test 
directorates and to the forward directorates as required. 


> Develop, modify and acquire technology to support the test 
requirements. 


> There are job descriptions, but nothing on how to support or manage 
stakeholders. 


> | have not seen anything on the functions of individuals or expected 
behavior to conduct within the organization. 
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Comments Received (cont.) 


@ Responsibilities 

> Responsible for providing technology support for all operational tests of 
the local directorates and for developing technology to support those 
tests. 

> Funds management, personnel management, direct test Support 
mission accomplishment, management of contractor response to 
tasks/requirements. 

> The only thing that comes close to listing responsibilities is the 
performance appraisal. 
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Comments Received (cont.) 


® Training 
> More technical courses needed, the limitation is funds available. 


> There is no certification process or training for M&S or Instrumentation 
Engineers. 

> We need also to stay current with DoD doctrine for unit behaviors and 
Capabilities. 

> Training should consist of communication, contracting, briefing 
(presentations), and organizational. 

> Certain courses are mandatory for all civilians, such as Prevention of 
Sexual Harassment, Drug & Alcohol Awareness, etc. However, there is 
no structured training program specifically pertaining to engineering, 
M&S, or program management skill areas required for the job. 


55 


Example 1 


Comments Received (concl.) 


@ Resources 


> 


Do not now have the required budget to properly accomplish our roles 
and responsibilities while executing our activities, but we think it is going 
to get better. 


The admin folks make us do our own admin tasks now. We arrange our 
own travel, submit our own vouchers, and edit our own documents. | 
don't know what they do or why they are here any more. 

Small office, inadequate storage space, little or no funding to purchase 
jump drives, etc. Most personnel probably purchase some equipment, 
such as jump drive storage devices out of personal funds. 

This organization may need more in terms of security assurance, 
DITSCAP, threat, data management and program management. 


Need better collaboration tools. 
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Follow-up email 


@ Director sent out a follow-up email 


@ Requesting the 9 that responded to the survey to prioritize the 
elements in order of importance 


5/ 


Example 1 


Follow-Up email (concl.) 


BCC Addressees - A few weeks ago you completed an Organizational Survey. | appreciate the time 
you took to respond to the questions and provide additional information. I'm now asking that you assess 
the importance to you of 20 topics related to the survey areas. Just put a 1-to-5 assessment of 
Importance to You in front of each of the 20 areas. You can send your responses to me or to Al 
Florence. 

1-low, 5-high What is the Importance to You of each of the following: 


. Published mission statement 

. Published vision statement 

. Published strategic goals 

. Identification of contractors responsible for products 

. Identification of customers for organizational products 
. Identification of key stakeholders for products. 


. Documentation of key activities linked to each employee 
. Documentation of key roles linked to each employee 


. Documentation of your responsibilities 

. Documentation of staff skills/experience required for each position 
. Training to accomplish your mission 

. Policy, procedures and actions for technology insertion 

. Published DoD Ethics Standards of Conduct 

. Communications and status meetings at division level 

. Policy regarding information sharing 

. Process to accomplish day-to-day activities 

. Detailed Procedures to support the execution of the processes 
. Documented standards for products and services. 

. Facilities including collocation of team 

. Resource to accomplish your mission and activities 


Jane Doe 58 


Example 1 
Priorities Individual Responding —— 1 


Published mission Individual Priorities —~> 3 


ie) 


Published vision 

Published strategic goals 

Identification of contractors responsible for products 
Identification of customers for organizational products 
Identification of key stakeholders for products 
Documentation of Key Activities linked to each employee 
Documentation of Key Roles linked to each employee 


Documentation of your responsibilities 
Documentation of staff skills/experience required for each position 


Training to accomplish your mission 

Policy, procedures and actions for technology insertion 
Published DoD Ethics Standards of Conduct 
Communications and status meetings at division level 
Policy regarding information sharing 


Process to accomplish day-to-day activities 
Detailed Procedures to support the execution of the processes 


Documented standards for products and services 


Facilities including collocation of team 
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Priorities vs. Executive Dashboard 


Priorities as voted Executive Dashboard 


6 | 3.33 y Mission 
10 3 R_ Vision 
10 3 R_ Strategic Goals 

5 | 3.44 B Contractors 

7 | 3.22 B Customers 

3.55 y Key Stakeholders 
3.2 © Activities 

10 3 R_ Roles 

6 | 3.33 R_ Responsibilities 

Ranking in —) 11 | 2.89 | © Staff 
Order of priorities 2 | 3.67 | G Training 

3 | 3.56 Y Technology Insertion 
12 | 2.78 0 Ethics 

7 | 3.22 QO Communications 
12 | 2.78 O Information Sharing 
14 | 2.56 R_ Process 

7 | 3.22 R Procedures 

9 |} 3.11 © Standards 
13 | 2.77 | O Facilities 

1 | 4.22 © Resources 60 


Example 1 
First Cut at Near/Mid/Long Term Activities 


@ Near Term 


> Resources (tools, budget) 


> 


Training 


@ Mid Term 


> 
> 
> 


> 
> 


Key Stakeholders 
Mission 

Vision 

Strategic Goals 
Roles 


@ Long Term 
> Communications/Information Sharing 


> Technology Insertion 


> 


Staff 


VV VV WV 


Responsibilities/Activities 
Staff (Skills/experience) 
Process 

Procedures 

Standards 


@ Ethics & Facilities received very low in priority 


|__Areport was complied and sentttothe rector | 61 


Example 1 


Received From Director 
Al, 


You can critique my near term plan. | sorted all items in priority sequence 
from high to low. Then | assessed the ratings for the high priority 
items. That led me to this conclusion: 


Three areas for improvement plan - near term: 


- Tools to accomplish your mission - highest priority item and low rating 
(I've asked the team leaders for input on resources needed by their team 
members. In about a month, I'll follow-up with information to 
individuals. Recent priority need in one area for ear protection headphones 
was met as urgent requirement. I'll use that as an example of how needs 
can be met.) 


- Documentation of your responsibilities - high priority item and lowest 
rating 
(Since having a standard operating procedure for each Organizational 
position is low priority, | understand the need to me clearly stating 
responsibilities for each person, not detailing how to do the work). 
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Received From Director (concl.) 


- Process and other policy documents - high priority item and lowest 
rating 
(Since having a standard operating procedure for each 
Organizational position is low priority, so our focus needs to be 
documenting Organizational-level processes, not focusing on how to 
do individual missions). 


My intent was to brief this action plan during last week's ‘all-hands' 
meeting. Had less than 50% attendance because of travel and ran 
out of time with other discussions. I'll now handle through a 
combination of Team Leaders' meetings and directorate email. 


Your critique welcomed. 


Jane 
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Introduction 

Dysfunctional/Stove Piped Organizations - Associated Problems 
A Solution - Organizational Analysis 

Organizational Structures 

Organizational Analysis Elements 

The Big Picture - How an Organizational Analysis is Conducted 
Organizational Analysis Example 1 

Organizational Analysis Example 2 

Summary 

Reference 

Questions/Answers/Discussion/Comments 

Contact Information 
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Example 2 


Conducted on a civil agency 

@ Done by interviewing only, no questionnaire 
> This method takes longer than sending out questionnaire 
> Is intrusive on the organization 


> Interview questions need to be open ended 


« If “Yes/No” questions asked they need to be followed up with request for 
supporting data such as: 
= An artifact = Amemo 
= A document = Meeting minutes 


@ Individuals interviewed 
> Executive management 
> Senior management 
> Supervisors 
> Key staff members 
> Key stakeholders 
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Example 2 


Approach 


Assess the current state of organization via research and 
interviews 
> Identify mission and vision 
> Analyze capability to meet the mission 
> Identify improvement opportunities 
> Make improvement recommendations 
@ Recommendations to help organization in defining: 
A single enterprise repository of policies, procedures, and standards 
A common high-level process view 
Required roles and responsibilities 


> 
> 
> 
> Key activities for interactions with major stakeholders 
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Example 2 


Organizational Elements 


@ MITRE identified ten major organizational elements or areas in 
which to assess the state of the organization, the elements are: 


V VV VV VV VV WV 


Mission, Vision, and Charter 

Strategic Goals and Activities 
Customers, Products, and Services 
Contractors, Products, and Services 
Organizational Structure and Facilities 
Roles and Responsibilities 

Staff and Training 

Policies and Process 

Procedures and Tools 
Communication 


67 


Example 2 


Questions Asked 


@ Key to questions: 
> E - questions for executives 
> M - questions for managers/supervisors 
> W - questions for worker bees 
> 


G - questions for anyone (some may not be appropriate for executives, 
you decide) 


@ When asking questions one may deviate from the canned questions 
as necessary 


@ The answers to questions may elicit additional questions to gather 
all required information 
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Questions 


@ Candidate questions for “Mission” 
> Would you please explain the mission of the organization? E, G 
> What do you think the mission of the organization is? E, G 
> Is the mission of the organization documented? G 
> Can you please provided us a copy? M 
@ Candidate questions for “Vision” 
Would you please explain the vision of your organization? G 
What do you think vision should be? G 
Is the vision of the organization documented? G 


Vv VV WV 


Can you please provided us a copy? M 

@ Candidate questions for “Charter” 

Would you please explain the charter of your organization? G 
What do you think the charter of the organization is? G 

Is the charter of the organization documented? G 


Vv Vv V WV 


Can you please provided us a copy? M 
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Example 2 


Questions (cont.) 


Candidate questions for “Strategic Goals” 
> Would you please explain the strategic goals of the organization? E, G 
> What do you think these goals of the organization should be? E, G 
> Are these goals documented? G 
> Can you please provided us a copy? M 
Candidate questions for “Activities” 


> Would you please explain the required activities to meet the strategic 
goals? G 


> What do you think these activities should be? G 
> Are these activities documented? G 
> Can you please provided us a copy? G 
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Questions (cont.) 


@ Candidate questions for “Customers” 
> Would you please tell us who the customers of the organization are? G 
> Which are internal or external to the organization? G 
> Who are the end users of the organization’s products? G 
> Who are your immediate customers? W 
> Are these customers described anywhere? G 
> Can we have a copy? M 
Candidate questions for customer “Products and Services” 


> Would you please tell us what products the organization produces and 
for whom? G 


> Would you please tell us what services the organization provides to Its 
customers? G 


> What do you think these products and services should be? G 


> Are these products and services documented and/or described 
anywhere? G 


> Can we please have a copy? M 
rail 


Example 2 


Questions (cont.) 


@ Candidate questions for “Contractors” 
> Would you please tell us who the contractors of the organization are? G 
> Who are the end users of the contractors’ products? G 
> Are there statements-of-work (SOWs) for these contractors? G 
> Can we have a copy? M 
Candidate questions for contractor “Products and Services” 


> Would you please tell us what products the contractors produce and for 
whom? G 


> Would you please tell us what services the contractors provide to their 
customers? G 


> What do you think these products and services should be? G 


> Are these products and services documented and/or described 
anywhere? G 


> Can we please have a copy? M 
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Example 2 


Questions (cont.) 


@ Candidate questions for “Organization Structure” 


Vv VV VV WV 


Do you have a diagram of the organization? G 

Is there text to support the diagram? G 

Are all stakeholders identified G 

Are responsibilities of all branches documented? G 

Are relationships to branches and other stakeholders documented? G 
Can you please provide us a copy? M 


@ Candidate questions for “Facilities” 


> 


Can you please tell us where the Project Management organization 
physically resides? G 


> Are individuals in close proximity to each other? G 


> Is there diagram of the facility? G 


> What are the room arrangements, furniture, and other facilities for 


items as computers and printers? W 
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Example 2 


Questions (cont.) 


@ Candidate questions for “Roles” 


> 


Vv Vv V WV 


Where are position descriptions for the roles of the organization 
documented and can we please have a copy? M 


Can you please explain your roll within the organization? G 

What are the functions that you perform on a day-to-day basis? W 
Are these documented? G 

Can we have a copy? G 


@ Candidate questions for “Responsibilities” 


Vv Vv V WV WV 


Can you please explain your responsibilities within the organization? G 
What duties do you perform on a day-to-day basis? W 

What outputs or outcomes are you responsible for? W 

Is this documented? G 

Can we have a copy? G 
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Questions (cont.) 


@ Candidate questions for “Staff” 


> Are the required skills and experience for the various roles of individuals 
documented? M 


> Can you please provide this to us? M 


> How do you ensure the staff members and project management have the 
required skills and experience for their roles? M 


> Could you please tell us what skills and experience you have to support 
your day-to-day duties? W 
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Questions (cont.) 


Candidate questions for “Training” 


> Do you have a training program for individuals to augment the skills 
required for their roles and responsibilities? M 


> Do you have a training schedule? M 

> Where are the training requirements for individuals documented? M 
> Do you maintain training records? M 

> Can you please provide us copies of these? M 


> What training do you receive to augment the skills required for your roles 


and responsibilities? G 
> Where is this required training documented? W 
> Do you have records of the training you have received? W 
> Can we please have copies of these records? W 
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Questions (cont.) 


Candidate questions for “Policy” 


> What high-level policies do you have for the activities and processes 
required to conduct the functions of Project Management ? M 


> Can you please provide them to us? M 
> What high-level polices do you follow to conduct your required activities? W 
Candidate questions for “Process” 


> What documented processes do you have for the execution of the day-to- 
day activities required to support your activities? G 


> Can you please provide them to us? M 
> How do you ensure that these processes are practiced as documented? G 
> Please explain how these processes are consistent and repeatable? G 


> Can you please tell us what processes exist to support your day-to-day 
activities? G 
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Questions (cont.) 


@ Candidate questions for “Procedures” 


> Do you have step-by-step documented procedures for the execution of 
the processes required to support staff and project management 
activities? G 


> Can you please provide them to us? M 

> How do you ensure these procedures are practiced as documented? M 

> Please explain how these procedures are consistent and repeatable. G 
Candidate questions for ”Tools” 


> Do you have a list of the tools that are used to support the day-to-day 
activities of the staff and project management? M 


> Can we please have this list? M 
> What tools do you use to support you day-to-day activities? W 
> What training do you receive on these tools? G 
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Questions (concl.) 
Candidate questions for “Communications” 


> 


Vv Vv VV WV 


Vv Vv V WV WV 


How do you ensure that individuals or Project Management have open 
and clear lines of communications with you, each other and other 
stakeholders? E, M 


How often are informational and status meeting held with the staff? G 
How often are informational and status reports published? G 
Can we please have some copies of these reports? M 


Do you have an open door policy for your staff to communicate with you? 
M 


Does your manager/supervisor have an open door policy for you to 
communicate with him/her? W 


Do you use this open door policy? Was it useful? W 

How often do you have teambuilding sessions? G 

Can you please explain if these teambuilding session are useful? G 
Do you have meeting minutes for status and teambuilding sessions? G 


Can we please have copies of meeting minutes? M 
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Interviews Conducted 


@ Two Executives 
@ Two Managers 
Four Supervisors 
@ Three Key Stakeholders 
@ General staff 
> Four sets 
> 2 to 5in each set 
@ Resulted in much valuable information 
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Organizational Analysis Report 


Section 1: Introduction - Defines the purpose of the assessment and the 
organization of this report 

Section 2: Executive Summary - Provides a high-level overview of the approach, 
findings, and recommendations. 

Section 3: Current State - Presents the results of the analysis reflecting the current 
state 

Section 4: Desired State - Presents recommendations to achieve the desired state 

Section 5: Implementation Plan - Presents the priorities, the plan, and schedule to 
develop and implement the recommendations 

Appendix A: Category Definitions - Presents detailed descriptions of the categories 
that were assessed 

Appendix B: Consolidated Comments - Presents the interview comments against the 
categories compiled and consolidated 

Appendix C: Interview List - Presents a list of the interviews that were conducted by 
organization the individual belong to, name of individual, and date 

Appendix D: Acronym List - Provides an alphabetical list of the acronyms used in this 
document and their definitions 

Appendix E: List of References - Presents a list of references to augment this report. 

Appendix F: List of Implemented Recommendations - Presents a list of items that 
have been developed and implemented 
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Will Look At 


@ Summary of Findings 

Current “As-Is” State (Executive Dashboard) 
@ Samples of Comments Received 

@ Samples of Summary Recommendations 

@ Samples of Detailed Recommendations 

@ Analysis Mapped to Recommendations 

@ Follow-up Working Meetings 

@ Sample “To-Be” Transition Strategy 

@ Sample of “To-Be’” Transition Plan/Schedule 
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Summary of Findings 


@ Mission, Vision and Charter 
> Mission and vision statements are not documented but some individuals 
have ideas as to what they should be 
> Charter not documented but some comments were received on what the 
charter might be 
Strategic Goals and Activities 
> Strategic goals are not documented although some individuals have ideas 
as to what they should be 
@ Customers, Products and Services 
> Customers, products and services are well Known but documentation, 
delivery schedules and oversight are generally missing 
@ Contractors, Products and Services 
> Contractors, products and services are well Known but documentation, 
delivery schedules and oversight are generally missing 
Organizational Structure and Facilities 
> A detailed organizational chart and organizational documentation does not 
exist 
> Staff is located on three different floors and most staff members have 
obsolete equipment 83 
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Summary of Findings (concl.) 


@ Roles and Responsibilities 
> Roles and responsibilities are in some cases known and being 
accomplished but are not documented 
> Some roles and responsibilities are not assigned or assignments are not 
known so some individuals take on additional responsibilities 
Staff and Training 
> Staff members have been adequate up to this point, for basic support of 
the web site, with the appropriate skills for their roles and responsibilities 
> Training is generally lacking, especially as new products and services are 
introduced and for project and product management 
Policies and Process 
> Policies are generally missing. Some processes exist but are not 
documented 
Procedures and Tools 
> Some procedures exist but are not documented 
> Some tools are being used but training on tools is missing 
@ Communications 
> Communications are generally good throughout the entire organization 
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Current “As-Is” State 


Mission (Yellow) 


[Nomission statementwiitendown SSS 
[Many suggestions that need tobe compledinto amissionsiatement | fei | | 


Vision (Yellow) 


[Novision statement written down 
[Many suggestions that need tobe compiled into avision statement | |G] | | 


Charter (Yellow) 


No charter written down Py | | ie 


Many suggestions that need to be compiled into a charter Pele) |] 


Strategic Goals (Yellow) 


No strategic goals written down 
Many suggestions that need to be compiled into strategic goals 


Activities (Yellow) 


Everyone understands the activities that are being conducted iB) | |] 
The activities are not documented in relation to the goals, roles and 
responsibilities they support 


Example 2 
Current “As-Is” State (cont.) 


Customers (Orange) 


Everyone knows who their customers are je) ff 
Documentation on customers’ information is minimal | | | | |R 


Customers’ Products and Services (Orange) 


Everyone knows what products and services are provided to their customers and 
are providing them 
Documented information on customers’ products and services is minimal and yyy 


processes are not established and followed for development and monitoring of 
Everyone knows who their contractors are G 


delivery schedules 
Contractors’ information is minimal and contractor monitoring is generally lacking |] | Ri 


Contractors (Orange) 


Contractors’ Products and Services (Orange) 


Everyone knows what products and services their contractors provide lol | | 
Information on contractors’ products and services is minimal and processes are not 
established and followed for development and monitoring of delivery schedules 
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Current “As-Is” State (cont.) 


An organizational structure is not documented and an organization diagram 
does not exist 


| The relationships to interfacing organizations are not documented =| relationships to interfacing organizations are not documented lalla 


Staff is located on different floors BECEE 
Some staff located in different building || | fol. 


Not everyone |Not everyone knows and understands theirroles and understands their roles [Not everyone knows and understands theirroles J | | 
In some cases roles have not been documented and some 
— are duplicating efforts 


Not everyone knows and understands their responsibilities TT | Rr 


In some cases responsibilities have not been assigned to 
individuals and some staff members are taking on responsibilities 


that they feel are not being accomplished 


Example 2 
Current “As-Is” State (cont.) 


Staff seem to have the required skills to perform their roles iG} | f- 


Responsibilities are somewhat in flux at this time, as content creation and 
publishing are transitioning to content owners 


A formal training program or training plans do not exist 
Very little training has been conducted; some feel they could use training on 
specific topics 


Policies generally do not exist Py | f | 
Policies for critical and essential activities, products and services do not exist} | | | | 


Processes are not documented but exist for some areas, such as content 
posting and investment decision, but missing in others, such as product, 
project, contract, and configuration management 
LL 


Some processes are executed but not documented 
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Current “As-Is” State (concl.) 


Some procedures exist but are not documented. Some are being developed 
for areas such as content posting and investment decision. Some 
are missing such as configuration management and quality assurance. 


A few procedures are loosely documented in guidelines 


Some tools exist for some activities 


Training on tools is generally inadequate 


Communications are generally good throughout the entire organization 


Some managers have an open door policy and they respond in a timely 
fashion to staff members 
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Samples of Comments Received 


@ Lack of Clarity in Roles and Responsibilities 


> Several people talked a lot about doing whatever it takes to get the job 
done, and they indicated that in the past they have gotten dinged at 
performance appraisal time because they were viewed as doing 
something that “wasn’t their responsibility”. 


> It sounds like sometimes it’s not clear whose job it is to do certain things, 
or those things just are not getting done, so people pick up those tasks to 
make sure they get done. 

> Fred’s group took over registration product from Sam, which was a huge 
help, but took away the clarity of roles. 

> The prime contractor should be integrator of all the pieces, but there were 
issues of ownership, and it hasn’t been clear who should make decisions. 

«> Plus, different parts of projects are owned by different groups, which 
further complicate communication and decision-making. 
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Samples of Comments Received (cont.) 


Lack of Training 


> The team members felt that it would be useful for them to attend 
technical forums to better understand requirements and what the users 
need to support development. 
« They indicated that they have not been allowed to attend many forums, or 
when they are allowed to go, they are designated as note takers. 
«» They seemed to feel that even when they get to go, their need for being there 
is not recognized, while the other group seems to be very involved in the 
forums. 


> The participants indicated that when product ownership was given to 
them, training on project management and oversight was lacking. 


«» They also had to work with contractors, and were often a daily contact for the 
contractors, but they weren’t really trained in contractor oversight — it was not 
clear to the team members what they should be pushing the contractor to do, 
as opposed to what they should be asking acquirer to do. 


« As a result, the contractor ended up driving the project. Incomplete 
requirements are due to the contractor constantly trying to limit the number of 
requirements and delaying some to later implementation, which seldom 
occurred. The contractor also has had a lot of turnover resulting in lost work 
and increasing the learning curve. 91 


Example 2 
Samples of Comments Received (onc. 


@ Lack of Tools 


> They do not have access to tools, like Rational Rose, except maybe in 
the lab, and some people do not have access to the lab, and do not have 
accounts to access needed software. 


> They feel like they are pushing the limits to try to get around all their 
obstacles, but feel they should not have to work so hard to get around 
these sorts of obstacles, and in some cases are bending the rules to try 
to get the job done, and fear repercussion. 


> After the requirements were baselined, they could not get access to view 
the latest requirements. 

> The team members felt that it would be useful for them to attend forums 
to better understand requirements and what the users need to support 
development. 
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Samples of Summary Recommendations 


@ Mission and Vision statements need to be created and published. 


Strategic Goals and a Charter to meet the Mission and Vision need to 
be created and published. 

@ An Organizational Structure that supports the Strategic Goals and 
Charter needs to be created and published. 

#@ Roles and Responsibilities that support the organization and the 
achievement of the strategic goals need to be reviewed and updated 
as the organization changes. 

> Clear roles and responsibilities need to be assigned to individuals and 
these assignments documented. 

Existing policies, processes and procedures need to be reviewed and 
updated. 

> New ones to support the roles and responsibilities and the achievement 
of the Mission, Vision and Strategic Goals need to be developed and 
followed. 
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Samples of Summary Recommendations cont.) 


@ Key relationships with other organizations need to be defined in a way 
that supports the achievement of the Mission, Vision and Strategic 
Goals. 

@ Electronic equipment and tools need to support the roles, 
responsibilities, policies, processes and procedures, and need to be 
upgraded or acquired. 

@ Individuals in the organization need to be located in closer proximity 
to each other 

@ A training program that supports all the above needs to be created 
and executed. 

@ Computing equipment is obsolete and Is affecting both moral and 
productivity. 

> State-of-the-art equipment would resolve this problem. 
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Samples of Summary Recommendations cont.) 


Staff 
> Skills are needed in the following areas: 
“Technology, including XML, web services, and content management 
“Project and program management 
“» Security 
“* Telecommunications 
@ Training 
> Training needs to include tools, processes and procedures, web site 
functions and capabilities, and policies and regulations 
> Staff should be trained on the technologies being used for the web site, 
including Vignette, WebTrends, Akami 
> Staff should be trained in project, program, and contract management 
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Samples of Summary Recommendations  conc1,) 


@ Procedures and Tools 


> Detailed step-by-step procedures need to be developed to support the 
implementation and execution of the processes. 
> Staff members need to have state-of-the-art electronic equipment and 
other tools to support their day-to-day activities. Some require laptop 
computers and some read/write CD capabilities. Some require project 
management tools. 
@ Communications 


> Team building sessions need to be held and assignments need to be 
documented for each staff member. 


> Effective communications will help staff members in understanding their 
roles and responsibilities and working as a cohesive organization. 
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Samples of Detailed Recommendations 
Role Performed 


Programmatic Oversight Develop Requests for Proposals (RFPs) and Statements of Work (SOWs) for 
contractors’ products and services, when necessary. 


Develop and monitor schedules for contractor products and services. 


Conduct technical and programmatic oversight of contractors’ activities that 
produce products and services. 


Review and approve contractor products and services 
Coordinate with Procurement on contract issues 

CCB Chair Serve as one of three co-chairs of the CCB 

Suggest agenda items, and approve final agendas 

Provide welcome and introduction at each meeting 

Steer group according to charter (education, communicationnot decisions) 


Technical Oversight Contribute to and review all RFPs, SOWs, etc., for technical content 
Participate in requirements definition for content management and portal 
Participate in status meetings and design reviews for portal and applications 
Coordinate with steering committee on technical issues for the site 


Technicallnvestgatlons Conduct frequent reviews and investigations into technologies for the portal 


Make recommendations for technology use or refreshment on the portal 
97 


Example 2 


Samples of Detailed Recommendations (cont, 


Performance Management Manage the performance of activities that support the roles and responsibilities 
in producing products and conducting services, both from the contractors and 
Program office perspective. Processes may include: 
¢ Product/Program/Project Management 

Contractor Management and Oversight 

Requests for Proposals 

Statements of Work 

Plans, Schedules & Budgets 

Requirements Management 

Reviews 

Configuration Management 

Quality Assurance 

Metrics 

Risk Management 

Earned Value (cost/schedule monitoring) 

® Transition to Support 


Investment Decision Determine whether a new or additional feature should be added theweb site. 
This includes conducting impact analysis by all stakeholders against, but not limited to 
the following: function, cost and schedule. Processes may include: 


e Baselining 
e Impact Assessment (function, cost, schedule) 
e Change Control 


Rapid Application Provide the capability to develop smaller in-house web capabilities faster and 
Development cheaper without sacrificing functionality and quality 


Example 2 


Samples of Detailed Recommendations (concl.) 


Application and Manage the application and content of the web sites. This 
Content Management includes providing guidance and technical and programmatic support to 
users and content providers of the web site. Processes may include: 


Content Analysis 
Content Design 
Content Implementation 
Content Consulting 


Web Usability Testing Conduct tests on the web site to determine the ease of using the 
site. Processes may include: 


Test Planning 
Test Procedures 
Test Conduct 
Test Reporting 


Web Statistics Gather and report statistics on web site usage and issues. Procedures may include: 


Gathering Statistics 
Graphing Statistics 
Analyzing Statistics 
Reporting Statistics 


Example 2 
Analysis Mapped to Recommendations 


The Mission, Vision, and Charter must first be defined, and must tie back to 
the Missions, Visions, and Charters of the higher level organizations 


®@ The Mission, Vision, and Charter help to define the Strategic Goals and 
activities that must exist and occur to meet the Mission, Vision, and Charter 


C The Mission, Vision, and Charter and Strategic Goals and activities then 
help to define what products and services will be offered to what customers, 
and which contractors’ products and services will be utilized to make those 

Sd 


offerings 

Once the products and services are defined, then the organization can be 
defined to deliver those products and services. A set of roles and 
responsibilities can be defined that tie back to the products and the Mission. 
Those roles can then be crafted into an organizational structure with certain 
needs for specific staff and training 


The organization will need policies, processes, procedures and tools to help 
them accomplish their mission in a reliable, repeatable manner. The higher 


level policies and processes must first be defined, and these will then drive 
out more detailed procedures and tools 100 


Example 2 


Follow-up Working Meetings 


Several follow-up brain storming meetings conducted involving: 
> Executive 
> Managers 
> Supervisors 
> MITRE representatives 
> Meeting moderator 
Examining 
> Mission 
> Vision 
> Strategic Goals 
> Other Areas 
> Next steps 
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Sample To-Be Transition Strategy 


@ This transition strategy will support the organization in establishing 
the desired “To-Be” state and in achieving the Strategic Goals: 


> Establish priorities 


«> Prioritize the categories and the implementation activities based on the 
importance and value added to the Mission and Vision statements and 
Strategic Goals 


¢* Develop and prioritize a transition “To-Be” plan into short-term, mid-term and 
long-term implementation increments based on the importance and value 
added to the Mission and Vision statements and Strategic Goals 
> Establish an implementation schedule 
«> For short-term implementation increments (less than 12 months) 


¢ For mid-term implementation increments (greater than 12 months, but less 
than 24 months) 


¢» For long-term implementation increments (greater than 18 months, but less 
than 36 months) 
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Sample To-Be Transition Strategy (concl.) 


> Obtain Funding and Resources for short-term implementation 
¢* Determine and obtain funding 
«+ Determine and obtain resources 
«> Investigate best practices for short term implementation categories 
= Investigate organization’s best practices 
= Investigate industry best practices 
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Sample To-Be Transition Strategy (concl.) 


@ Implement best practices for short term implementation categories 
> Use best practices if available and usable 
> Tailor best practices, if necessary 
> Develop new practices, if required 
> Deploy practices 
> Deploy on trial basis 
> Fine tune practices 
> Deploy for general use 
> Use, maintain and improve practices 
@ Use Executive Dashboard to track improvements over time 
@ Repeat for mid-term implementation 
@ Repeat for long-term implementation 
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Sample To-Be Transition Plan 


Mission and Vision Statements —=—SsS—=“~*~“‘—~‘“dSC |x 
[Strategic Goals and Charter _—~=S=~=“*‘“‘*~*S*‘dCSdY 
[Organizational Structure ——~S~C~—~“—*~“—*~‘“ | 
Roles and Responsibiities documented andassigned | 4 | x |i 
Relationships defined with other key organizations | 5 |x [=| 
Staffand Skils Defiton ——SSSSSC*iS | x 
[Acquisition of needed equipmentandtools —S«d 7 | x 
Critical policies, processes and procedures «| —=S*dSSCStC~“‘dSSCSC~irtCS 
[Performance Management _—=SSS=~<~idC 
[investment Decision —Ss—C—C—~“s*~“*‘“‘“s~‘id Px 
[Application and Content Management ——«dt aod] SiC 
TusabiltyTesting—SSCSC=C—~iCa SP Sid 
Statistics Si aed 
Cer 
raining Program ——SC~SC~“S~S na SP Sid 
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Sample To-Be Transition Schedule 


Establish Priorities 
Establish Schedule 
Funding and Resources 
Investigate Best Practices 
Develop Practices 

Deploy Practices 

Use Practice 


Midterm Implementation 


Long term Implementation 


12 Months 
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Where Are We? 


¢$¢+ OO 


¢¢¢ ¢ 


Introduction 

Dysfunctional/Stove Piped Organizations - Associated Problems 
A Solution - Organizational Analysis 

Organizational Structures 

Organizational Analysis Elements 

The Big Picture - How an Organizational Analysis is Conducted 
Organizational Analysis Example 1 

Organizational Analysis Example 2 

Summary 

Reference 

Questions/Answers/Discussion/Comments 

Contact Information 
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Summary 


@ Before starting Process Improvement or other initiatives, it is best to 
determine if the organization is ready for change 
> If not ready, change becomes very difficult 
@ In many cases organizations are dysfunctional and stove piped 
> They reorganize thinking that this solves their problems 
> This only buries them further into the abyss 


@ An organizational analysis may be necessary to surface the root causes 
of the problem and break down barriers 


@ Fixing problems at their roots and breaking down barriers 
> Allows organizations to function as cohesive and seamless units 
> Provides for easier process improvement and other initiatives 
> Facilitates success 
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Where Are We? 


YH 


Si A A A A A A a A A 


Introduction 

Dysfunctional/Stove Piped Organizations - Associated Problems 
A Solution - Organizational Analysis 

Organizational Structures 

Organizational Analysis Elements 

The Big Picture - How do we Conduct an Organizational Analysis 
Organizational Analysis Example 1 

Organizational Analysis Example 2 

Summary 

Reference 

Questions/Answers/Discussion/Comments 

Contact Information 
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Agenda 
Introductions and Logistics 


. Why? 
Why should an organization attempt to calculate the ROI of CMMI- 
based efforts? Is ROI the right way to show financial value? 
. When? 
When should these ROI calculations be performed? When can you 
compare ROI results across organizations? 
. What? 
What information is needed in order to calculate ROI? What 
information is inappropriate? 
How? 
How do organizations calculate ROI? How can they do it better? 


. Wrap-Up 
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Carnegie Mellon 
Software Engineering Institute 


How often have you heard this....? 


What is the value of process improvement to our 
organization? 


Why CMMI? 
What should we focus on? 
How much is this going to cost? 


What's in it for me? 
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Overview 


This tutorial is intended to be a basic introduction to 
calculating the costs and benefits of CMMI-based process 
improvement 


We'll discuss estimating, tracking, and validating ROI over 
the course of a CMMI improvement effort 


You'll be introduced to several ways to calculate ROI, and 
the pros and cons of each 


Implementation approaches, hints and tricks will be 
introduced to help you really make this work! 


——s Carnegie Mellon 
~—|-__ Software Engineering Institute 


Logistics 
Workshop Time/Duration 


Rest Rooms 
Breaks 
Smoking Rules 
Phones 


Messages 
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ROI can be calculated many ways 
There is no single, official “definition” of ROI — it can be 
calculated in many ways. 
Some of these include: 
¢ Benefit/Cost ratio (typically considered ROI) 
¢ Net Present Value (NPV) 
¢ Internal Rate of Return (IRR) 
¢ Payback Period (PP)/Break Even Point (BEP) 


A complete ROI analysis includes all of the above. 


== Carnegie Mellon 
Software Engineering Institute 
The Value of CMMI Improvement 


Average software organizations spend 65% or more of their 
engineering dollars addressing quality problems! 


This means that only 1/3 of the organization is actually 


creating something. 
Leading software organizations can reduce this Cost of 


Quality to 40% or less, resulting in 50% or more gains in 


productivity. 
Numerous CMM case studies conclusively demonstrate the 


value that can be achieved. 
A rapidly growing number of CMMI case studies are 


duplicating & exceeding these results. 
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7 Examples 


Reduced cost of poor quality from over 45% to under 30% 

¢ 2:1 ROI over 3 years (Siemens Information Systems Ltd, India) 
$2.1 Million in savings in hardware engineering processes 
(Reported under non disclosure) 
Reduced software defects per million delivered SLOC by over 
50% compared to defects prior to CMMI 
(Lockheed Martin Systems Integration) 


Reduced defect rate at CMMI ML5 approximately one third 


compared to performance at SW-CMM ML5 
(Lockheed Martin Maritime Systems & Sensors — Undersea Systems) 


Avoided $3.72M in costs due to better cost performance 
¢ As the organization improved from SW-CMM level 4 to CMMI 
level 5 (Raytheon North Texas Software Engineering) 


2.5:1 ROI over 1S‘ year, with benefits amortized over less than 6 
months (reported under non disclosure) 


Carnegie Mellon 
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7 Thales Research & Technology 


CMM data from another Thales Unit used by Thales 
Research & Technology as part of rationale to begin 
PI with CMMI. 


a Results measured by a THALES Unit © 


for Software impact on cost deviation 


Improvements in: 


Average Cost Variance - 
reduced by 20% 4 % Project Distribution 


On average late to 
acceptance - divided by 
“4 

Number of defects 
during Customer 
acceptance - divided by 
2 


Cost of customer 
acceptance - reduced by 


ry bes i 
0  % Cost deviation | 
THALES 


Getting Started with Process Improvement Using the CMMI® Carol Marsh, Patrick Vigier. 
ESEPG 2003. 
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Northrop Grumman IT 
Appraised at CMMI ML 5 in December 2002 


Results Improvements in: 
* met 25+ milestones in a row 
* earned a rating of “Exceptional” in every 

applicable category on a formal Contractor 

Performance Evaluation Survey 


Hours Invested: 124 in Defect Prevention (CAR) 
Hours saved: 1650 hours (15 hours per defect) 
ROI: 13:1 


Integrating PSPS and CMMI® Level 5. Gabriel Hoffman, Northrop Grumman IT . May 1, 2003 
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Accenture 


Transition SW-CMM to CMMI ML 3 nae 
« May 2001 to May 2002 
¢ Transition Time: 1149 person hours 
Key Content 
Measurement and Analysis 
DAR = TS, RM, Change Control 
IPPD = visions, OEl 
Generic Goals 


Results Improvements in 


¢ ROI: 5:1 (for quality activities) [orga 


Innovation Delivered. CMMI® Level 3 in a Large Multi-Disciplinary Services Organization. 
Bengzon, SEPG 2003 
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Some Caveats! 


No financial or accounting knowledge is assumed 


The tutorial is not intended to be comprehensive, some 
topics are presented at a high-level only 


In some cases, nuances of financial accounting and RO! 
determination are not addressed 


We'll do our best to get you started without burying you in 
details! 
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Audience 


Executive/leaders of organizations seeking to understand the 
costs/benefits of CMMl-based process improvement and how to 
quantify them 


Process improvement/EPG personnel seeking ways to 
communicate more effectively to senior management about the 
costs/benefits of CMMI-based process improvement 


Personnel in organizations starting down a CMMI-based process 
improvement path 


Project managers and engineers who need information to make 
better decisions about their project work. 


Personnel in higher-maturity organizations seeking to transition to 
the CMMI, or implement higher-maturity process areas 


7 
Calculating CMMI-Based ROI: Why When, What, and How? 
November 15*, 2004 
© 2004 by Carnegie Mellon University 


Carnegie Mellon 
Software Engineering Institute 


= Carnegie Mellon 
~—|!___ Software Engineering Institute 


7 Introductions 


Tutorial leader introductions 


Participant introductions 
« Name 
*« Company 
¢ Position 
¢ Expectations 
- What do you want to get out of the tutorial? 
- Do your expectations match the tutorial agenda? 
- How can we best address your concerns? 


——s Carnegie Mellon 
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~ Tutorial Approach 


Interactive Presentation 
Examples 
Exercises 


Questions and Answers 


Participate!!! 
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2—Why? 


To develop the business case for process 
improvement 


Need to communicate to decision makers who 
understand the language of money, not CMMI 


iii. ROI is the only way to show financial value 


Sponsored by the U.S. Department of Defense 
© 2004 by Carnegie Mellon University 
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Business Context 


The first question to answer is “what is the business context 
in which we’re undertaking a CMMI effort?” Is it because 
we want to: 


* Do things better, faster, cheaper? 
¢ Increase contract award fees? 

¢ Secure more contracts? 

¢ Improve product quality? 


We’re going to focus on the quality, project 
management, and productivity implications of CMMI 
process improvement 
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System Development Costs 


Costs incurred to develop the product 


Examples: 


¢ Requirements development 

* Design 

¢ Product development + unit 
test 

* Project management 

¢ Engineering infrastructure 

¢ Purchased components 


What isn’t on this list? 


Carnegie Mellon 
Software Engineering Institute 


Cost of Quality (CoQ) Concept 


Developed by J.M. Juran and applied successfully by 
companies like Toyota Motor Corporation to achieve 
competitive advantages through the development of better 


quality products 


CoQ represents all costs associated with poor quality 
Prevention Costs 


7 Costs to Achieve Quality ¢ 
Appraisal Costs 
CoQ i 


Internal Failure Costs 


Costs Due to Poor Quality ¢ 
External Failure Costs 


Source: Juran’s Quality Handbook 
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Cost of Quality Appraisal 


Objectives 


¢ Quantify quality costs in the 
language management 
understands — money 

* Identify major opportunities 
for quality cost reduction 

* Identify opportunities for 
reducing customer 
dissatisfaction and threats to 
product saleability 

¢ Expand budgetary and cost 
controls 

¢ Stimulate improvement 
through publication 


Improve ability to deliver 
more to the business/market 
Reduce the functionality gap 
versus the competition 
Provide executives with 
reasons to invest in 
engineering improvement 
CoQ appraisals are an easy 
way to understand where 
money is spent and to 
gauge the effectiveness of 
an organization 


Carnegie Mellon 
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Internal Failure Costs 


Costs associated with defects that are found prior to transfer 
of the product to the customer 


Examples: 


¢ Design corrective action 
« Design re-reviews 
¢ Purchased component 


« 2nd and subsequent 
integration testing iterations 
« 2nd and subsequent system 


corrective action 
¢ Purchased component re- 
test 
¢ Defect reporting/tracking 
* Defect fixing 


testing iterations 
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External Failure Costs 


Costs associated with defects that are found after the 
product is shipped to the customer 


Examples: 


« Next release defect rework ¢ Lower marketplace 
(maintenance) perception 

* “Re-engineering + Loss of pricing power 

* Technical support personnel =, Lost sales 

¢ Product returns 

« Lawsuits 

¢ Contract penalties 

¢ Lost customers 


Carnegie Mellon 
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Graphical Representation of CoQ 


Total Cost of Quality 


Most projects are in 
this area 


Prevention & Appraisal Costs 


0 Costof Quality 


n # of Product Defects 


Spending a little more here 
translates into much less 
cost later 
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Prevention Costs 
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Costs incurred to keep failure and (product) appraisal costs 


to a minimum 


Examples: 

* Quality planning 

* Quality assurance 

* Configuration management 

¢ Supplier capability 
assessments 

* Quality training 

* Component/software reuse 


Carnegie Mellon 
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(Product) Appraisal Costs 


Costs incurred to determine the degree of conformance to 
quality requirements 


Examples: 


¢ Purchased component 
testing 

¢ Defect reporting/tracking 

¢ Test automation 
systems/software 


« Requirements reviews 

* Design reviews 

* Code reviews 

¢ CM tools 

« External process 
appraisals/audits (CMMI, 
ISO, SPICE, etc) 

* Process improvement efforts 


¢ First iteration integration 
testing 

First iteration system testing 

¢ User acceptance testing 
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Relating CoQ to Productivity 


Decreasing an engineering organization’s Cost of Quality 
directly impacts productivity: 


¢ Fewer defects mean less time testing and reworking the 
product 

¢ Less time testing means a shorter cycle time 

¢ Shorter cycle time means more products can be 
developed in similar time frames 

¢ Fewer defects also mean a smaller customer support 
staff — these resources can be applied to new product 
development 


==. Carnegie Mellon 
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SW Example: 65% More Productivity 


Average [®ttieucas Design Test 
4.5 Months 6.6 Months 9.4 Months 
Leading Gui i 
(ela Requirements Design Development Rewark 
4.0 Months 6.0 Months 6.8 Months 


Leading organizations Leading organizations 
spend 20% less on software deliver 80% less defects to 
development effort and production, resulting in 
schedule, and 50% less on significantly less rework 
testing costs! and 45% more productivity! 


20% 45% 


Average organizations spend 30% of project resources on testing, leading organizations spend 15% 
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Leverage 


Process improvement has a leverage Higher Revenue and 


effect beyond just quality and productivity Profits through 
increases More Marketable 
Improved Products 


Current Capability 
Capability 
CoQ 


Productivity 


Carnegie Mellon 
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Communication 


Executives, senior management, and other decision 
makers don’t use CMMI language 


$ Their language is money 
$ Decision making is done in the context of money 


Communicating the importance and effectiveness of 
CMM-based process improvement 

Champions must communicate in the language of 
executives — money 
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Business Case 


¢ Resources are scarce (especially $money$) 
¢ Itis impossible to execute on all opportunities available 
to the organization 


¢ Executives want an “apples-to-apples” comparison of the 
various investment options available 


Incorporating ROI analysis into a proposed or 
existing CMMI effort allows for comparison of 
efforts across seemingly disparate projects 


And, the ROI of CMMI can be quite high 
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Provide Focus and Tracking Capability 


Performing an ROI analysis up-front (estimation) can 
* focus the organization on concrete business objectives 
for CMMI process improvement, 


* clarify how improvement efforts map to corporate 
objectives, 

* establish expectations about results 

¢ establish a baseline to track performance 


Multi-year process improvement efforts need to maintain 
the business focus by continuously tracking 


$ the associated actual costs 
$ the related benefits 
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Drive Better Results 


Desirable Results 


Average 
Results 


Process 
Improvement 
Benefits 


Payback Period Break-even Point 


Process 
Improvement 


Costs Undesirable Results 


CMMI Cost/Benefit 
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Difficulties in Using ROI 


Differing perceptions of ROI 

Buy-In 

Access to Data 

Calculating benefits is tough 

Consistently measuring ROI over time is tough 


Organizations are resistant to measurement 
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Discussion 


What other 
difficulties have 
you 
experienced in 
determining 
CMMI ROI? 
How did you 
overcome 
them? 
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A Process for Determining ROI 


Determine 
Effort Scope/ 
Current Needs 


v 
Estimate/ 
Determine 
Effort Costs 

v 

( ‘ Estimate/ 
Determine 
Effort Benefits 


Estimate/ 
Calculate ROI Track actual 


J costs/benefits to 


determine actual 
Present ROI 


Results to 
Management 


Continued process 
improvement 


Approve and execute effort 


Version 1.1 
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3 — When? 


i. Before 
ii. During 
iii. After 


Sponsored by the U.S. Department of Defense 
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Before 


Initially, it is VERY important to understand the potential 
value of a CMMI-based improvement effort to: 


¢ Clearly understand the estimated costs associated with 
the effort 

¢ Demonstrate the potential business value 

¢ Provide a comparison against competing projects 

¢ Establish a baseline for ongoing tracking and oversight 

¢ Establish the expectations that benefits require time to 
develop, particularly for senior management 

¢ Communicate with decision makers 


At this time, other organizational indicators such as quality 
measurements, schedule adherence, customer satisfaction, 
etc. should be baselined. 
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7 During 


In order to track and maintain visibility, ROI should be 
tracked on an ongoing basis (perhaps every 6-12 months) 


At the same time, intermediate indicators such as quality 
measurements, customer satisfactions surveys, etc. are 
tracked to determine the impact of improvement activities 


Doing so demonstrates that the effort is on track from a 
cost perspective, and continues to keep the focus on the 
effort. 


Additionally, early wins are communicated 
unambiguously in financial terms — something 
executives like to see. 
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After 


After a particular effort is completed, the ROI should be 
calculated to fully understand the true value of the effort. 


This is needed to evaluate the financial effectiveness of the 
effort, create lessons learned, and set the stage for driving 
continuous improvement and institutionalization. 


This forms the basis for the implementation of higher 
maturity process areas that support fact-based decision 
making. 
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In other words.... 


continuously, 
throughout the entire effort! 


Implementation of the Measurement and Analysis process 
area early in your adoption effort will facilitate this. It will 
also support your organization’s comparison to applicable 
benchmarks as well as modeling and simulation. 
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4 — What? 


All applicable direct costs should be 
identified 

ii. As well, all direct benefits should be 
accounted for 


iii. Indirect costs and benefits shouldn’t be 
part of the analysis as they are difficult to 
attribute solely to the effort 
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What information is needed..... 


In its most basic definition ROI = A comparison of the costs 
and benefits of a process improvement effort across a 
specific organizational scope and time span. 


You need to understand: 


* Scope of the analysis 

¢ Appropriate time horizon for analysis 
¢ Relevant costs 

¢ Financially quantifiable benefits 


Remember, for a proper ROI analysis, all cost and 
benefit information needs to be calculated in terms 
of dollars and cents. 
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Process Improvement ROI Model 


Non-Financially Financially Difficult to Attribute 
Quantifiable Quantifiable Secondary 
Primary Benefits Primary Benefits Business Benefits 


Reduced Cost of Improved Customer 
Quality Resulting Satisfaction 
in Productivity Improved Employee 
Increases Satisfaction 
* more features/ Lower Training 
functionality Costs 
delivered for Increased Sales/ 
the same price Market Share 
* same features/ Improved Award 
functionality Fees 
delivered fora 
lesser price 
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Scope of Analysis 


There are many ways to scope an ROI analysis. 
¢ At the: 
- Organizational level 
- Project level 
- Process level 
¢ Examples include: 
Moving from CMM to CMMI 
Introducing all practices associated with a particular PA 
Implementing specific practices 
Achieving a specific maturity or capability level 


¢ ROI calculations/equations won’t change due to 
scope. 


——s Carnegie Mellon 
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Time Horizon 


Over how much time should an ROI analysis be 
performed? 


Organizational - Generally, organizational efforts must 
show net value over a 2-3 year time period and 
unambiguous incremental value in shorter time frames. 


Project/process - must show value in a shorter period of 
time. 


It is probably unrealistic in most circumstances to 
demonstrate meaningful and measurable benefit in less 
than 3 months time due to: 

¢ Need to develop/implement processes 

¢ Train organizational units 

¢ Institutionalize the improved processes 
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Scope: Exercise 
At this point, we’ll begin working though our fictitious 
example of: 
¢ Organizational CMMI improvement effort 
¢ 150 person engineering organization at maturity level 1 
¢ Looking to reach maturity level 2 in 18 months 


¢ No EPG or QA in place 
¢ Experiencing a Cost of Quality of 65% 


In the exercise, we'll utilize an ROI calculation worksheet 
that guides the type of information we need to consider. 


First off, we need to define the scope of the effort 


== Carnegie Mellon 
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What Costs and Benefits? 


What costs do you need to track? 
What benefits are you expecting to realize? 


When will costs and benefits be incurred? 
How will these come together to generate ROI? 
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Key Factors 


Generating Cost & Benefit Data, you need to be aware of: 


¢ Consistency 
- in gathering over time 
- in matching costs and benefits 
¢ Clear & detailed operational definitions 
¢ Clearly understanding incremental costs and benefits 
¢ Costs and benefits defined in financial terms 
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ect People Costs of CMMI Improvement 


Costs of Process Definition / Improvement 


Process definition & Interviews, meetings 
documentation 


MSG, EPG, TWGs Attendance in training 
Management of the effort Process implementation 
Consulting Process institutionalization 
Additional personnel Process adherence 
Project team diversion Assessments/appraisals 


Communication Rewards and recognition 


Generally counted as salary & benefit costs associated 
with people’s time 
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Direct Non-People Costs of CMMI 
Improvement 


Costs of Process Definition / Improvement 


Tools PAL, Measurement 
Repository 


Training 
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Indirect Costs of CMMI 
Improvement 


Costs of Process Definition / Improvement 


Loss of organizational/project 
team productivity 
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7 Other Costs to Think About 


Rework due to poor quality 
* Prior to shipping 
° After shipping 


Poor project cycle time 


Lack of schedule predictability 


Rather than consider these expenses on the “cost” 
side of the ROI analysis, we’ll consider improving them 
to be part of the “benefits” side. 


Carnegie Mellon 
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Costs: Exercise 


¢ Organizational CMMI improvement effort 

¢ 150 person engineering organization 

¢ Looking to reach maturity level 2 in 18 months 
¢ No EPG or QA in place 

¢ Experiencing a Cost of Quality of 65% 


Which costs should we plan and account for? 
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Direct Benefits 


Reductions in: Improvements in: 
Defects Process adherence 


Rework Quality 


Effort/overtime Product features 


Schedule cycle time 


Product costs 


Production costs 
Support costs 
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Indirect Benefits 


Reductions in: Improvements in: 


Training costs Employee 
satisfaction/retention 


Customer 
satisfaction/retention 


Revenue 


Award fees 


Market share 
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Discussion 


Most direct e 
process ogo? COsTs 
improvement 

benefits can be BENEFITS 
financially 

quantified in 

terms of 

productivity 

increases 
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7 So, How Do We Measure Productivity? 


The definition of Productivity is the number of outputs 
delivered for a certain number of inputs. 


For systems engineering, the inputs are typically people- 
related. 


The outputs however, can vary significantly, and are 
difficult to compare against each other: 

¢ Missile system 

* Copy machine 

* Cell phone 

¢ Function points or SLOC (for software) 


The easiest way to help understand the notion of 
productivity in an engineering organization is Juran’s 
Cost of Quality concept. 


Carnegie M 
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Measuring Non-Financial Benefits 
As discussed, there are many non-financial benefits with 
different ways to measure them. 


It is important to measure non-financial benefits on an 
ongoing basis as part of the effort in order to keep 
focus on other indicators of progress or value. 


Keep in mind that measuring these in non-financial terms 
doesn’t directly impact the ROI calculation. 
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Using Benchmarks to Estimate 


To aid in determining what costs and benefits may be 
expected, consider the use of industry benchmarks. For 
example: 


* Gartner research shows software organizations spend 
3% - 5% on CMMI process improvement efforts 

¢ CMM maturity level 1 organizations typically experience 
a Cost of Quality of 65% 

¢ CMM maturity level 3 organizations typically experience 
a Cost of Quality of 40% 


Baselining the organization and using benchmarks to 
compare against helps in building a credible ROI 
calculation. 
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- Using Simulation to Estimate 


Engineering process simulation models focus on the 
dynamics of engineering development and maintenance. 


Modeling and simulation of the significant features of 
engineering processes facilitates experimentation, answer 
“what-if” questions, and perform sensitivity analysis. 


By modeling an organization’s current engineering 
processes and simulating improvements to them, costs and 
benefits can be better estimated and determined. 


Example: would the cost of source code inspections add 
value if the input source code had an abnormally high 
degree of errors? 
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Benefits: Exercise 


¢ Organizational CMMI improvement effort 

¢ 150 person engineering organization 

¢ Looking to reach maturity level 2 in 18 months 
¢ No EPG or QA in place 

¢ Experiencing a Cost of Quality of 65% 


Benchmarks indicate that an organization operating at 
maturity level 2 can reduce its Cost of Quality to 50% 


How can we quantify these benefits? 
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Levels of Data Fidelity 


The level of detail you go to in an ROI analysis will depend 
on the size of the effort 


Multi-million dollar CMMI efforts in larger organizations will 
most likely require a greater degree of data fidelity thana 
smaller, more tactical solution on just one project. 


Understand the level of fidelity needed to make a credible 
ROI analysis. Don’t blow your chance with a shallow case 
if decision makers are expecting more details. 


“Test run” your assumptions and analysis with peers 
and other managers before presenting the final results. 
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5 — How? 


There are several ways to calculate ROI 


Using multiple methods provides for a multi- 
faceted view and allows for better decision 
making 
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ROI can be calculated many ways 
There is no single, official “definition” of ROI — it can be 
calculated in many ways. 


Some of these include: 
¢ Benefit/Cost ratio (typically considered ROI) 


¢ Net Present Value (NPV) 


¢ Internal Rate of Return (IRR) 
¢ Payback Period (PP)/Break Even Point (BEP) 


A complete ROI analysis includes all of the above 
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_ Match Cash Flows and Don’t Double 


Count 
Costs must align appropriately with benefits 
¢ Don't include costs or benefits that don’t relate to each 
other 
¢ Beware of “one-time” costs that should really be incurred 


over the entire ROI analysis time period 
¢ Don’t scope the effort’s costs to a specific project but 


count benefits organizationally 


No double counting 
¢ For example, you can’t count the value of an avoided 


defect and the productivity that’s improved as a result 


33 
Calculating CMMI-Based ROI: Why When, What, and How? 


November 15', 2004 
© 2004 by Carnegie Mellon University 


—— Carnegie Mellon 
—=—} Software Engineering Institute 


Carnegie Mellon 
Software Engineering Institute 


Benefit/Cost Ratio 


Benefit/Cost Ratio (typically considered to be ROI) is simply 
the ratio of benefits enjoyed by a certain effort divided by 
the costs incurred. i.e. 150% 


Pros 
¢ Fairly easy to compute 
¢ Provides a “one number” method of comparing projects 


Cons 
¢ Doesn't incorporate the Time Value of Money (TVM) 


¢ No way to know the dollar magnitude of the return 
¢ Don’t know how long the return took to achieve 
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Benefit/Cost Ratio 


Year 0 Year 1 Year 2 Total 
Costs $300 $100 $100 $500 
Benefits $0 $500 $500 $1000 
Return -$300 $400 $400 $500 


Benefit/Cost Ratio in this example would be 2:1 or 200% 
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Time Value of Money (TVM) 


Benefit/Cost ratio, while valuable, doesn’t take into account 
the concept of the Time Value of Money (TVM). 


Simply stated, the value of a dollar is less in the future than 
it is now. 


Question: If risk-free interest rates are 10%, how much 
would you pay me now for $100 in 3 years? 
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Time Value of Money (TVM) 


Answer: $75.13 


Year 0 Year 1 
Principal $75.13 $82.64 
Interest $7.51 $8.26 
Total $82.64 $90.91 
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Discounted Cash Flow (DCF) 


To aid in corporate decision-making, the TVM is captured in 
Discounted Cash Flow (DCF) analysis. 


In DCF analysis, future cash flows are discounted using 
either: 
¢ The company’s weighted average cost of capital 


(WACC) 
¢ A “hurdle rate” consisting of the company’s cost of 


capital adjusted by a risk premium appropriate for the 
type of project being analyzed 


——s Carnegie Mellon 
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Discounted Cash Flow (DCF) 


This risk premium can be significant (30%+) in some cases! 


Year 0 Year 1 Year 2 
Principal $45.52 $59.17 $76.92 
Interest $13.65 $17.75 $23.08 
Total $59.17 $76.92 $100 
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Net Present Value (NPV) 


Net Present Value (NPV) takes both cost and benefit cash 
flows and discounts them appropriately using the 
company’s cost of capital or hurdle rate. 


NPV is described as a dollar value, allowing for seemingly 
dissimilar projects to be compared against each other. 


Pros 
¢ Articulates the dollar magnitude of the return 
¢ Provides a “one number” method of comparing projects 
¢ Incorporates TVM 


Cons 
¢ Not as easy to compute 
¢ Don’t know how long the return took to achieve 


= Carnegie Mellon 
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Net Present Value (NPV) 


With a 10% hurdle rate, our example looks like: 


Year 0 Year 1 Year 2 Total 
Costs $300 $100 $100 $500 
Benefits $0 $500 $500 $1000 
Return -$300 $400 $400 $500 
Disc Return -$300 $363.63 $330.58 $393.94 
(NPV) 
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Internal Rate of Return (IRR) 


IRR determines the intrinsic interest rate that summarizes the 
value of a series of cash flows. 0% indicates “break-even”, 
positive percentage indicates a positive return while a negative 
percentage indicates a negative return. 


Essentially determines the discount rate that results in NPV=0 


Pros 
¢ Provides a “one number” method of comparing projects that is 
independent of the company’s WACC or hurdle rate 


Cons 
¢ No way to know the dollar magnitude of the return 
¢ Don’t know how long the return took to achieve 
¢ In unusual cases, may provide multiple answers 
« Not as easy to compute 
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Internal Rate of Return (IRR) 


In our example, the IRR is 100% 


Year 0 Year 1 Year 2 Total 
Costs $300 $100 $100 $500 
Benefits $0 $500 $500 $1000 
Return -$300 $400 $400 $500 
Disc Return-$300 $200 $100 $0 
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Payback Period/Break Even Point 


Determines the amount of time it takes for cash inflows to 
equal cash outflows, resulting in the break even point 


Pros 
¢ Indicates how quickly a project turns profitable 


Cons 
¢ No way to know the dollar magnitude of the return 


¢ In unusual cases, may provide multiple answers 


Carnegie Mellon 
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Payback Period/Break Even Point 


In our example, the PP/BEP is Year 1 


Year 0 Year 1 
Costs $300 $100 
Benefits $0 $500 
Return -$300 $400 
Net Return -$300 $100 
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Exercise 


¢ Organizational CMMI improvement effort 

¢ 150 person engineering organization at maturity level 1 
¢ Looking to reach maturity level 2 in 18 months 

¢ No EPG or QA in place 

¢ Experiencing a Cost of Quality of 65% 


How do the various ROI calculations turn out for our 
exercise? 


Carnegie Mellon 
Software Engineering Institute 


Pittsburgh, PA 15213-3890 


6 — Wrap-up 


Sponsored by the U.S. Department of Defense 
© 2004 by Carnegie Mellon University 


Version 1.1 CMMI RO! Tutorial - slide 80 
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Implementation Issues 


ROI analysis won’t necessarily improve a CMMI effort’s 
results — you still have to execute! 


In any process improvement effort, you should find ways to 
leverage organizational change best practices to overcome 
resistance to change. 


Determining and tracking ROI as well as other 
measurements keeps the focus of the effort on the 
business benefits 


Use the Measurement and Analysis process area early to 
put the focus on the business benefits from the start 


5) 2004 by Carnegie Mellon University Version 1.1 CMMI RO! Tutorial - slide 81 
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Discussion 
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What Did You Learn? 


Why? 

i. To develop the business case for process improvement 

ii. Need to communicate to decision makers who understand the language of money, not 
CMMI 


iii. ROlis the only way to show financial value 


When? 


i. Before, During, and After 

What? 

i. All applicable direct costs should be identified 

ii. As well, all direct benefits should be accounted for 


iii. Indirect costs and benefits shouldn't be part of the analysis as they are difficult to 
attribute solely to the effort 


How? 

i. | There are several ways to calculate ROI 

ii. | Using multiple methods provides for a multi-faceted view and allows for better decision 
making 


Case Study 


We went through a case-study where we estimated CMMI-based process improvement costs 
and returns over a 3 year time period, and explored the various ways of calculating the 
value of the effort over the 3 years. 


Carnegie Mellon 
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A Process for Determining ROI 


Determine 
Effort Scope/ 


Current Needs Continued process 


Y improvement 
Estimate/ 
Determine 
Effort Costs 


q 


Estimate/ 
Determine 
Effort Benefits 


t 


Estimate/ 
Calculate ROI! 


t 


Present 
Results to 
Management 


4 


Approve and execute effort 
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Parting thoughts: 
What’s Next for You?! 


Determining the monetary value of CMMI-based process 
improvement may appear to be difficult. 
But it’s not that hard. 
¢ It does take disciplined effort 
¢ You may have to build or buy the necessary 
organizational capabilities & resource capacities 


And it’s extremely important. 
¢ Making informed decisions about how to invest scarce 
resources 


¢ Sustaining commitment for disciplined process 


improvement 


== Carnegie Mellon 
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Thank You! 


Rolf W. Reitzig — cognence, inc. 
- Resident SEI Affiliate 
- Rolf Reitzig@cognence.com 


Dennis R. Goldenson — SEI 
- dg@sei.cmu.edu 


Diane Gibson — SEI 
- dig@sei.cmu.edu 


Mark R. Cavanaugh — IBM Global Services 
- Resident SEI Affiliate 
- markrc@us.ibm.com 
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= Process Goals 


Proven Approach 


Process Improvement Infrastructure 


Process Execution 


Applying Techniques 


= Questions 
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= Build a process that is 


= Adaptable to changing user needs 
a Maintainable to implement changes rapidly 


= Flexible to support a solid architectural framework 


= Modularized components 
= Reusable components 
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= Build a process that is NOT 


= Cumbersome to understand (shelfware, big honkin’ binder) 


= Circumvented when schedule and cost pressures affect a 
program 


= Difficult to maintain 


=m Changes require redesign 
= Fixing one error propagates more errors 
= No programmer wants to be assigned to the team 


Operationally Oriented; Customer Focused 


4 Process Goals Continued 
& 


U.S. AIR FORCE 


= Build a process that supports 


= Daily needs of the engineers and managers 
= Various levels of expertise 
= Multiple disciplines with a unified process 


= Ownership by the engineers 
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= Define the Level of Detail 
= Establish an approach that meets user experience levels 


= Define the Design Components 
= Cartography, Step Action Table (SATs) 
and Decision Tables (DTs) 
= Artifacts 


= Define the Environment 
= Processes 
= Tools 
= Analysis Methods 
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= Cartography is a graphical representation designed for 
highly experienced engineers that 


= /mproves the business objectives 


= Teaches others how to apply the 
organization’s processes 


= Designs and implements /nnovative 
technical solutions 
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= Step Action Table (SAT) is a textual representation 
designed for moderately experienced engineers that 


= Applies the business objectives 


= Executes the organization’s processes 


= Designs and implements proven technical 
solutions 
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= Decision Table (DT) is a tabular representation with 
tailoring options to include templates and checklists 
designed for inexperienced engineers that are 


= Understanding the business objectives 
= Learning the organization’s processes 


= Assisting with the design and 
implementation of technical solutions 
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= Represents high-level overview of process activities to 
include a link to process-related information such as 
policy, training, frequently asked questions, Quality 
Assurance (QA) tips and Best Practices 


= Supports point and click process navigation using web 
pages 


= Requires Microsoft Visio to implement t 
to create the graphical symbols 
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Requiremerp 


Mana 
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= Defines step-by-step details 
of the process activities 


= Supports point and click process 
navigation using web pages 


= Requires HTML and Java languages to implement web 
pages and the automatic numbering for process steps 
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Requirements Management Step Action Table 


(as of April 2006) 
Step | Action | Responsible | 
| Verify Entry Criteria 
| 1 |Verify that Project initiation has been completed or system requirements allocated to Project 
software have been modified Leader(PL) 
2  |Record/review the project purpose and for Standard Software Process (SSP) projects, PL {| 


Scope, Goals, and Objectives in the Project Tracking Form (PTF) Description 


section, as needed 


3  |Acquire the Software Requirements Specification (SRS) Package ? and record initial PL 


data and remaining data as tasks below are completed using the Requirements 
Checklist or an approved group-specific Requirements Checklist, and referencing the 
Requirements Compliance Checklist 


4 |Record/review the following data in the PTF Schedule Breakdown section for the PL 
Requirements Management and Project Planning phases: 


Requirements as a critical milestone or not 
Actual start date 
Adjusted effort and date estimates (may use calculate button in PDB to 
acquire adjusted estimate date based on the actual start date) 
e Development Team Members 


Measurements 


30 |Referencing the SRS Package, record the total number of initial requirements in the Requirement |PL 
Volatility Metric in the Project Metrics sheet of the Automated Metric Analysis Tool (AMAT) 


Verify/record actual hours in SERTS under the Requirements lifecycle activity, tailoring decisions 
in the PTF and requirements completion date in the Schedule Breakdown section of the PTF 


| 
i 


i 
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= Defines criteria and action for each critical process 
decision (i.e., when y exists then perform x) to identify 
tailoring options that meet the unique needs of each 
process user 


= Supports point and click process 
navigation using web pages 


= Requires HTML to implement web pages 
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N V 
4 Approach - Design 
Cksiebee Sample DT Component 


Software Requirements Specification Package Decision Table 


(as of April 2006) 


| When | Then 
| Software Requirements Specification (SRS) Package exists | Acquire and modify SRS Package, as applicable 


SRS Package does not exist, and Customer or First Level Supervisor (FLS) / Group Acquire one of the following to complete the SRS Package: 


Leader (GL) does not require a formal requirements document 


e The Software Requirements Specification Form (SRSF), or 
e Anapproved SRS Database that captures all data items listed 
in the SRSF 


SRS Package does not exist, and Customer or FLS/GL requires a formal Acquire the Requirements Traceability Matrix (RTM) Template and 
requirements document one of the following to complete the SRS Package: 


e The Condensed SRS (CSRS) Template, or 
e The SRS Template 
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= Definitions 


= Clarify terminology based on the organization’s processes 


= Checklists 


= Derived from a template 


= Identify detailed questions to enhance 
the effectiveness of the process 


= Focus on product and process quality 
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= Forms 


= Provides a unified method to collect 
data consistently 


= Identifies minimum data requirements 


= Focuses on capturing the right data to 
measure progress and quality 
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\ y Approach - Design Artifacts 
<P Plans 


U.S. AIR FORCE 


= Plans are 
= Derived from a specific template 
(Requirements Specification, Test Plan) 


a Utilized to Identify formal 
information required by 
the customer 
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Process Overview 


| Process Policy Training FAQ. QA Tips Best Practices 
General - FAQ (QA Tips ‘None 
‘Acceptance Test Policy Training FAQ |QA Tips ‘None 
Integration Test Policy |Training FAQ |QA Tips ‘None 
Peer Review Policy Training FAQ |QA Tips ‘None 
Process Development Policy Training ‘None None ‘None 
Project Tracking and Oversight Policy Training FAQ | JA Tips ‘None 
Rapid Implementation Process Policy |Training ‘None ( JA Tips ‘None 
‘System Test Policy Training FAQ |QA Tips Best Practices 
Technology Change Management Policy Training FAQ None ‘None 
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Requirements Management Process Best 
Practices 


1. OSP v5.0 
Artifact Type: Software Requirements Specification Form 
Description: 


This artifact 1s a good example of a completed SRSF with clearly 
defined requirements. 


Link: Artifact 
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Requirements Management Process FAQ 


1. OSP v5.0 4/24/2005 

As a new project leader in the requirements phase of the project, | 
wasn't sure if | had to do this step or not. Is it only if you are in the 
Project Planning phase or later? 


The PMF is required during Requirements to capture your initial 
requirements, your estimated/actual effort, and peer review defects. 


2. OSP v5.0+ 9/18/2006 
In the Training Requirements section of the PTF, can the PL modify or 
remove a skill from the team skill set? 


Yes you can modify or remove a skill from the team skill set. To 
delete a skill select the skill you wish to modify, this should 
highlight that skill, and click the delete button to remove. 
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= Web-based Processes 


=» HTML Processes 


= Allows for centralized access and 
easy navigation 


= XML Configuration Files 
= Allows for easy modifications of 
changing data and implementation of 
business rules 
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= Development of In-House Tools 


= Standard Engineering Request Tracking System (SERTS) - tracks 
change requests as well as estimated and actual hours expended 


= Process Database Tool (PDB)- centralizes data, captures information 
to track progress and provides real-time insight into status 


= Automated Metric Analysis (AMAT) - extracts data from PDB and 
SERTS to provide a means to track project goals against 
organizational goals and perform quantitative analysis on 
performance 


= Shared Knowledge Provider (SKP) —- provides access to process 
improvement tools and project data via a centralized web page 
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a Statistical Analysis Methods 


= Statistical Process Control with moving range to allow for 
analysis to determine if the processes are stable and under 
control 


= Mini tab tool provides numerous charts and graphs to aide in 
data analysis 
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= Produce Process Champions 


= Believe their individual words and actions ultimately improve the 
project performance and build a more positive work environment 


= Display a constant willingness to help others understand, utilize and 
improve the processes by effectively communicating the direct 
benefits to the project 


= Transform the engineer’s frustrations into positive energy and focus 
towards building a better organization 


= Seek to continually add value by improving disciplined processes 
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= Executive Steering Committee (ESC) 


= Senior Management (SM) 


= Standard Enterprise Process Group (SEPG) 
= Project Support Office (PSO) 
Technical Work Gro TWG | 
a up (TW) _ 
= User Group (UG) 


= Target Group (TG) 
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= Executive Steering Committee (ESC) 


= Membership includes 3 Senior Executives and 5 Senior 
Managers 


= Identifies process improvement approach and allocates 
funding and resources 


= Provides oversight to the Organization Standard Process 
(OSP), policies, charters, Process Improvement Plan (PIP), 
and budget 


= Supports ESC Meetings bimonthly to review progress of 
planned versus actual tasks 
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= Senior Management (SM) 


= Membership includes 5 Senior Managers 
= Guides and directs activities to achieve the strategic goals 


= Supports SM Meetings bimonthly 
review SEPG status 


- Addresses resource, budget, 
and training issues 
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= Standard Enterprise Process Group (SEPG) 


= Membership includes 7 to 10 Group Leaders and the 
Organization Quality Assurance (QA) Manager 


:- Facilitates the definition, execution, and improvement of 
policies and processes 


requests and addresses action plans 
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= Standard Enterprise Process Group (SEPG) 


= Oversees Technical Work Group, Project Support Office, User 
Group and Training activities 


:- Performs Causal Analysis and Resolution, Organizational 
Process Performance, Quantitative Project Management, and 
Organization Innovation and Deployment activities 


= Provides training courses to engineers 
and all levels of management 


= Reports status to ESC and SM 
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= Project Support Office (PSO) 


= Composed of 6 permanent and 1 rotating position 


:- Performs QA activities, training management, and process 
maintenance V 


= Maintains and creates applications 
supporting process improvement 


= Provides support to the SEPG 
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= Technical Work Groups (TWG) 


= Composed of engineers representing each 
target group 


- Develops processes and accompanying 
artifacts as defined by a charter 


a “— Group (UG) 


= Composed of representatives from all groups 


= Resolves process issues at the engineering level 
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= Target Group (TG) 


= 
- Adheres to Organization Standard Process 
© 


- Addresses Compliance Issues 


= Submits Change Requests 


= Reports Project Status 


- Attends User Group Meetings 


= Supports Technical Work Groups 
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Ghai eb Overview 


= Process Development Process 


= Cartography 

= Step Action Table 

: Artifacts 

- Supporting Processes 


- Evaluating Processes 
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Process Development | Process Overview 


Requirements 
List 
[initial] 


Develop/ 
Analyze Update 
Process Process 
Requirement: Components 


Process 
Artifacts 
[reviewed] 


Process 
Artifacts 
[initial] 


Verify Work 
Products 


New Process required 


Schedule 
[initial] 


Lesson 
Learned 
Form 
[draft] 


: Training 

Pilot Proces: Pxecul Database 
Training 

[updated] 


Feedback 
[draft] 


Measurement: 


April 2006 
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Cartography symbols 


Artifact 
[state] 
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Entry Criteria defines conditions that must be met to 
initiate the process 

Activity Box defines action that results in the 
creation/modification of an artifact to the next higher 
state 

Artifact is the direct result of an activity taking place 
and displays an associated state as the artifact 
progresses within the development cycle 

Process measurements are collection points for 
process data 

Dashed lines indicate the component Is optional. 
Activity boxes, artifact boxes and measurement 
components can be optional 
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Use predefined cartography symbols 


Include entry criteria, activity box, artifact box and measurement 
at a minimum 


Include no more than 7 activity boxes per process 
Start Activity Boxes with a verb in the description 
Display optional components with dashed lines 


Link to process-related information (policy, poe best 
practices, frequently asked questions and QA tips) 


Must have a last updated date 
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= Process Development Process Steps 


= Verify Entry Criteria 

= Analyze Process Requirements 

= Develop/Update Process Components 
= Verify Work Products 

= Pilot Process 


= Execute Training 


= Measurements 
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= Verify Entry Criteria 


= Project Leader (PL) verifies a Standard Engineering 
Request Form (SERF) is received stating a new 
process must be developed 


= Analyze Process Requirements 


= PL reviews existing process artifacts based upon current 
requirements 


= PL reviews the Lessons Learned Forms (LLFs) to gain insight on 
process and management lessons learned for process development 


= PL identifies resources required (i.e., tools, person hours, hardware, 
etc.) and provides estimated hours for updating/developing the 
process to the applicable Configuration Control Board (CCB) and the 
First Level Supervisor (FLS)/Group Leader (GL) 
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= Develop/Update Process Components 


= PL develops/updates the following process artifacts according 
to the Process Development section of the Standards and 
Style Guide: 

= Cartography 

Step Action Table (SATs) 

Forms 

Checklists, as required 

Decision Tables (DTs), as required 

Templates, as required 


= PL updates the Standard Definitions, as required 
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= Verify Work Products 


= PL verifies the process artifacts are complete and correct 


= PL provides the process artifacts to the CCB and FLS/GL for 
review 


= PL updates the process artifacts based on the CCB and 
FLS/GL feedback 


= PL submits the process artifacts to CCB for piloting 
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= Pilot Process 


= Configuration Control Board (CCB) selects project/group to 
pilot the process artifacts 


= CCB defines the piloting period start date and completion date 


= CCB trains the Project Leaders (PLs) and Development Team 
Members (DTMs) on the process being piloted 


= CCB solicits feedback from piloting project/group after the 
piloting period has ended and provides to the PL 
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= Pilot Process 


= PL updates the process artifacts based on the piloting 
feedback and generates lessons learned 


= PL submits the final process artifacts and the lessons learned 
to CCB for baseline approval 


= CCB submits finalized process artifacts and lessons learned 
to the SEPG and the QA Managers, if the process needs to be 
incorporated into the OSP 


Operationally Oriented; Customer Focused 


\ yi Process Execution - ; 
*e* Step Action Table Continued «& 


U.S. AIR FORCE 


= Execute Training 


= CCB executes the Training 
Process with affected individuals 


= Measurements 


= PL, CCB verifies/records actual hours in Standard Engineering 
Request Tracking System (SERTS) under the Implementation 
lifecycle activity to develop, review, update and train the 
process 


Operationally Oriented; Customer Focused 


4 » 
YF Process Execution — 
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U.S. AIR FORCE A rtifacts 


= Requirements List 


Schedule 


Process Artifacts 


= Feedback 


= Lessons Learned Form 


= Updated Training Database 


Operationally Oriented; Customer Focused 
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Supporting Processes “=” 


= Training Process 


4 4» 
YF Process Execution — 
<3 


U.S. AIR FORCE 


= Executed by the Organization Training Manager to 


= Identify training needs 
= Schedule classes 
= Track attendance 


= Acquire feedback on training effectiveness 


SSS ——  ——————— ———————————————————— a 
Operationally Oriented; Customer Focused 


4 » 
YF Process Execution- _ 
<3 e A ). 
U.S. AIR FORCE Evaluating Processes SOS 
= Capture the estimated and actual effort expended to 
define, implement, review and test the new process .. 


= Capture the resources (number of people, tools, 
training, etc.) utilized on the new process 


= Capture number of post release defects 
associated with the new process 


Operationally Oriented; Customer Focused 


AX fF . . 
YF Process Execution - a. 
«3° Evaluating Processes Continued So od 


U.S. AIR FORCE 
= Capture the number of compliance issues encountered 
while executing the Process Development Process 


FY05 Average Noncompliance Issues By Phase 


Initiation, 0.82, 5% 


PMR, 1.45, 8% 
1 


-— Requirements, 3.67, 20% 


Planning, 2.53, 14% 


Acceptance, 1.94,11%  —}— 


\ Design, 0.67, 4% 


System, 1.25, 7% Implementation, 1.97, 11% 


Operationally Oriented; Customer Focused 
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U.S. AIR FORCE 


Applying Techniques 


Operationally Oriented; Customer Focused 


\7 Applying Techniques 


U.S. AIR FORCE 


= Demonstrate Shared Knowledge Provider 
(SKP) Functionality 


=# Show links to applications 
= Show structure of the menu bar 
= Show process assets 


= Show point and click navigation of the processes 


Operationally Oriented; Customer Focused 


\7 Applying Techniques 


U.S. AIR FORCE 


= Execute Process Definition Exercise 


= Apply process and techniques learned to create a 
process as a group 


= Expand process to address the enterprise level 


SSS ——  ——————— ———————————————————— a 
Operationally Oriented; Customer Focused 
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Operationally Oriented; Customer Focused 
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Acronyms 


AMAT - Automated Metric Analysis Tool 
CCB - Configuration Control Board 
CG - Communications Group 

CM - Configuration Management 

DT - Decision Table 

DTM - Development Team Member 
ESC - Executive Steering Committee 
FLS - First Level Supervisor 

GL - Group Leader 

LLF — Lessons Learned Form 

OSP - Organization Standard Process 
PDB - Process Database 


Operationally Oriented; Customer Focused 
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YW 
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Acronyms 


PIP — Process Improvement Plan 

PL — Project Leader 

PSO - Project Support Office 

QA - Quality Assurance 

SAT - Step Action Table 

SEPG - Standard Enterprise Process Group 
SERTS - Standard Engineering Request Tracking System 
SKP - Shared Knowledge Provider 

SM - Senior Management 

TG — Target Group 

TWG - Technical Work Group 

UG — User Group 


Operationally Oriented; Customer Focused 


{ 2 
Sd References 


U.S. AIR FORCE 


= Gamberini, Rudy, Reuters Inc., “Process 
Cartography: A Graphical Approach to Process 
Definition”, SEPG 2000. 


= Olson, Timothy G., Quality Improvement 


Consultants (QIC), Inc., “How to Define Processes 
in Expert Mode”, SEPG 2000 
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Team Software Process 


The Team Software Process (TSP) Is a engineering 


development process originally developed for software 
teams. 


TSP addresses common engineering and management 
issues (the same ones addressed by CMMI). 

e cost and schedule predictability 

¢ productivity and product quality 

e process improvement 


USP 
¢ truly empowers teams and team members 
¢ is acomplete, mature, “operational” process 
¢ provides immediate and measurable results 
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Improved Predictability 


Effort and schedule deviation 
are dramatically improved. 


Source: CMU/SEI-2000-TR-015 


160% 


140% + 
120% + 
100% + 
80% + 
60% + 
40% + 
20% + 


0% 
-20% 


Average Schedule Deviation - Range 


——_——_= 


Pre TSP/PSP With TSP/PSP 
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100% + 
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60% + 
40% + 
20% + 


0% 
-20% 


Average Effort Deviation - Range 


Pre TSP/PSP With TSP/PSP 
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Improved Productivity 


A nine person TSP team from the telecommunications 
industry developed 89,995 new LOC in 71 weeks, a 41% 
improvement in productivity. 


A TSP team from the commercial software industry, 
developing an annual update to a large “shrink-wrapped” 
software product, delivered 40% more functionality than 
Initially planned. 


A TSP team within the DoD, developing a new mission 
planning system, delivered 25% more functionality than 
Initially planned. 
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Improved Quality 


Defects/KLOC 


An analysis of 20 
projects in 13 
organizations showed 
TSP teams averaged 
0.06 defects per 
thousand lines of new or 
modified code. 


Approximately 1/3 of : Level 1 Level 2 Level 3 Level 4 Level 5 TSP 
these projects were 
defect-free. 


Source: CMU/SEI-2003-TR-014 
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Accelerated Process Improvement 


TSP addresses or supports 
most of the capabilities er 
expected of a project team 
through CMMI Level 5. 
Level 4 
It provides either a “starting CMMI Maturity 
point” or a “next step”. — as 
Using TSP as a starting point, 
three organizations have 
advanced from ML1 to ML4 in 


less than 3 years. B Directly Addressed 
B Supported 

O Partially Addressed 
O Not Addressed 

@ Unrated 


Level 2 


50% 100% 


Percentage of SPs 
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TSP Results: NAVAIR AV-8B 


Mar. 2000 
Oct. 2000 
Jan. 2001 
May 2001 


June 2001 


Fen 2002 


June 2002 
Sep. 2002 


Began current CMM-based improvement 
effort (now a CMMI-based effort) 

Began PSP/TSP introduction sequence 
First TSP team launched 

CBA-IPI: CMM level 2; 3 KPAs satisfied at 
level 3; level 4/5 observations on TSP 
Received draft of CMM-TSP gap analysis 
(levels 2 and 3 only, minus SSM and TP) to 
help guide improvement efforts 

Received late-model gap analysis (including 
TP at level 3 and levels 4 and 5) 

Launched second TSP team 

CBA-IPI: CMM level 4 (16 months from L2!) 


See Crosstalk, Sep. 2002, “AV-8B’s Experiences Using the TSP to Accelerate SW-CMM Adoption,” Dr. Bill 
Hefley, Jeff Schwalb, and Lisa Pracchia, and Crosstalk, Jan. 2004, “The AV-8B Team Learns Synergy of EVM 
and TSP and Accelerates Software Process Improvement” 


PA -> 


Specific Goal 1 
SPinw 
SP1.2 
SP1.3 
SP1.4 
SPHES 
SP1.6 
SPiey 


Specific Goal 2 
SP2.1 
SP2.2 
SP2.3 
SP2.4 
SP2.5 
SP2.6 
SP2.7 
SP2.8 


Specific Goal 3 
SP3.1 
SP3.2 
SP3.3 
SP3.4 
SP3.5 


Specific Goal 4 
SP4.1 
SP4.2 
SP4.3 


PA -> 
Generic Goal 2 


Generic Goal 3 
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AV-8B CMMI ce Look” Profile 


RM [| RD [| TS | Pi | VE [VAL] cm [PPQA] MA [CAR] DAR] OE! [OPD] OPF] OID | oT [opp] PP [PMC] iPM | QPM[ SAM |RSKM_IT_| 


peepee te ry 


Fl 
Fl 


RM [RD [ TS [ Pi | ve | VAL] CM [PPQA] MA [CAR] DAR] OE! [opp] opF[ oiD [| oT [opp] pp | PMc] IPM | QPM | SAM [RSKM|_IT_| 
s sess) Sse ssa) Eas] 
a ) Ss Ss Ss Ss NR NR Ss Ss NR Ss Ss Ss Ss Ss Ss Ss 3 
LEGENDS 
Practices Goals 
Fully Implemented or Satisfied [SS satistied 
| Lt Largely Implemented Unsatisfied (Goals) 
| PI [Partially implemented | NR_|Not Rated . 
RAMEE Not implemented Source: NAVAIR 
| NR |Not Rated 
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NAVAIR P3-C Journey 


May 2002 


Process Action 
Teams (PATs) 


Documenting 
SSA 
Processes 


Process 
Improvement 
Group Kick-off 


February 2002 


Defined Web 
Requirements 


oe 
Sant og 
May 2004 


scaAmpis“ 


- Risk Management 


Source: NAVAIR 


- Measurement & Analysis 
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Improved Quality of Work Life 


“A more disciplined process allowed me to do a better job, and 
allowed me to balance my job with other aspects of my life.” 


“This project ended up a lot less stressful than other projects.” 


“Promotes a less stressful environment. Can track that the 
project is on schedule. Fewer defects are seen positively in the 
organization.” 


“It is nice to be associated with a project that had few defects.” 


“| liked the level of detail that went into initial plan, and the 
constant awareness of the schedule. Allowed us to make 
adjustments as the project went on, instead of waiting for a major 
milestone.” 


“It was nice that management finally allowed the team to create 
the schedule.” 
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Adoption 


Organizations that are using, piloting, or preparing to pilot 


the TSP. 


ABB 

ABC Informatica 

Activision 

Advanced Information Services 


Advanced Maturity Services, Inc. 


Alan S. Koch Consultants 
Ambient Consulting 
AMRDEC 

Boeing 

Centre De Investigacion En 
Matamaticas 

Census Bureau 

CQG, Inc. 

CRSIP / STSC / DRAPER 
Davis Systems 

DOE / Los Alamos 

DOE / Naval Reactors 
DPC Cirrus 

Dynamics Research Corp. 
EDS 

Halex Associates 

Heath Solutions, Inc. 


Helsana 

Honeywell 

IBM 

Intuit* 

lomega 

1.Q. Inc. 

KPMG 

L. G. Electronics 
Lockheed Martin / KAPL* 
LogiCare 

Los Alamos National Laboratory 
M/A-Com Private Radio Systems, Inc 
Magellan Navigation* 
Microsoft* 

Motiva 

NASA Langley 
NCR/Teradata 

NCS Pearson 

Northern Horizons 
Northrop Grumman 
Oracle* 

Prodigia S.A. de C.V. 


PS&J Consulting / 

Software Six Sigma 

QuarkSoft 

Respironics 

Rockwell Collins 

SAIC 

Samsung SDS 

Siberlink 

STPP, Inc. 

STSC 

Trilogy 

TYBRIN Corporation - Air Logistics 
University of Alabama / Huntsville 
University of Queensland 

US Army / AMRDEC 

US Navy /| NAVAIR* 

US Navy / NAVOCEANO* 

US Navy | NAVSEA* 

Xerox 


*Organizations we are currently 
working with 


Carnegie Mellon 
Software Engineering Institute 


TSP for Systems Engineering 


NAVAIR and other organizations have discussed the 
possibilities of adapting TSP for systems engineering use 
for several years. 


Late in 2005, an effort was launched to extend TSP 
practice to systems engineers working in NAVAIR 
organizations, beginning with those that have had success 
using TSP for software development. 


Several organizations, including at least one within 
NAVAIR, are forging ahead with their own TSP 
adaptations. 
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Building High-Performance Teams 


TSP builds high-performance teams from the bottom-up. 


Team 
SUT Cellars 


Teaming 
ST CHESS 


Carnegie Mellon Universit 


2) 


Goal setting 

Role assignment 
Tailored team process 
Detailed balanced plans 


Process discipline 
Performance measures 
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OJUFeVinvaaatelarele(siparcialmsy aii 


; Engineering - 14 


Carnegie Mellon 
Software Engineering Institute 


Personal Software Process? 


The PSP Is a process designed for individual use that 
applies to structured personal tasks. 


PSP builds the teaming skills required for the TSP. 


With PSP, developers learn how to use a defined process 
and how to measure, estimate, plan, and track their work. 


This leads to 
e better estimating, planning, and tracking 
e protection against over-commitment 
¢ a personal commitment to quality 
¢ personal involvement in process improvement 
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PSP-TSP Process Evolution 


TSP 


Team development 


PSP2 


Code reviews 
Design reviews 


PSP1 


Size estimating 
Test report 


PSPO 


Current process 

Time recording 

Defect recording 
Defect type standard 
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PSP Improves Performance 


Estimation accuracy 
¢ fewer underestimates 
* more accurate estimates 
¢ estimates balanced around 
Zero 


Quality 
e yield improves by 2X to 3X 
¢ fewer defects in unit test, 
integration test, system test 
¢ COQ is flat or reduced 


40 


20 


ewe -100% 0% 100% 


20 


-200% -100% 0% 100% 


20 


-200% -100% 0% 100% 
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PSP Quality Results 


Defects Per KLOC Removed in Compile and Test 


120 
110 


100 


Gees §=—Viean Compile + Test 
——*— __ PSP Level Mean Comp + Test 


Mean Number of Defects Per KLOC 


Program Number 
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Non-Software Disciplines 


Many software-intensive projects have significant non- 
software components in terms of 

e requirements and test 

¢ support activities 

e customer deliverables 


The ways that these “other” activities are planned, staffed, 
and managed are reflected in organizational structure. 
e separate departments for systems engineering, test, 
documentation, etc. 
e often depends on the size of the organization and the 
size of the typical project 
¢ multi-disciplinary teams 
¢ matrixed project teams 
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Introduction to Personal Process 


SEI teaches a two-day class, /ntroduction to Personal 
Process, which begins the individual quality journey by 
raising the issues of size measures and process and 
defect definitions for intellectual work other than software 
development. 


It makes both economic and technical sense to extend the 
formal definitions of such work so that it may be planned 
and tracked with TSP methods. 


NAVAIR has been a leader in adapting PSP and TSP to 
non-software work, and is actively engaged with SEI to 
formalize this work. 
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Process Improvement for “Others” 


Applying TSP practices to other disciplines besides 
software engineering can be relatively straightforward. 
e many teams are already doing it successfully 
e based on CMM originally, which was based roughly on 
Crosby’s five-level model of the manufacturing quality 
journey 
e planning and tracking mechanisms are not software- 
specific 
e size and defect definitions (by default) are rooted in the 
software-specific examples from PSP training! 


In order to adapt PSP for use by other disciplines, size 
measures and defect definitions must be addressed. 
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Size Measures 


For a size measure to be useful, it must be 

¢ useful for planning 

e precisely defined 

¢ directly countable in an intermediate or final product 
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Defect Definitions 


A defect is anything in an interim or finished product that 
must be changed for the product to be used as intended. 


Defects in test procedures, requirements analyses, 
specifications, or user documentation can all adversely 
affect a customer’s use of the delivered product. 


Defect definitions must make sense to the people who 
must correct them. 


Defect correction is sometimes called rework. 
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Building High-Performance Teams 


TSP builds high-performance teams from the bottom-up. 


Goal setting 


Team Role assignment 


SLUT Celrare) Tailored team process 
Detailed balanced plans 


Process discipline 
Performance measures 
Skills Estimating & planning skills 


OJUFeVinvaaatelarele(siparcialmsy aii 


Teaming 


Carnegie Mellon 
Software Engineering Institute 


Team Management Framework 


The TSP team management framework helps the team 
meet their planned commitments by providing support for 
e team communication and coordination 
¢ project tracking and status reporting 
° requirements management 
¢ change management 
e risk management 


Team members gather data and manage their personal 
plans. 


These data are consolidated at the team level and used by 
the team to manage the team’s plan. 
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TSP Base Measures 


Quality 


Schedule 


Source: CMU/SEI-92-TR-019 
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TSP Project Tracking 


Enter 
Week Task 
Completed 


Fah k=) a BY=y (2X04 by 
ohVA Oxo) an) eLeyat=vals 
and Phase 


Enter 
Size by 
foxeyanl eLearn 


© 2003-06 by Carnegie Mellon University 


Version 1.0 


Product 


Summar 
y Task Status 


Engineer A 


Updated Team and Team Task 


Engineer Task, Schedule, FeValo motor arsve lel (=) 
Valo M@LUr-VI19YMmm ad cleats Summary 


; Schedule 
Quality Status 
Summary Engineer A 
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Tracking with TSP Measures 


The TSP base measures can be combined to provide a 
number of derived measures for managing projects. 


TSP Derived Measures 


Estimation accuracy (size/time) 
Prediction intervals (size/time) 
Time in phase distribution 
Defect injection phase 
distribution 

Defect removal phase 


distribution 

Productivity 

%Reuse 

%New Reusable 

Cost performance index 
Planned value 

Earned value 

Predicted earned value 


Defect density 

Defect density by phase 
Defect removal rate by phase 
Defect removal leverage 
Review rates 

Process yield 

Phase yield 

Failure cost of quality 
Appraisal cost of quality 
Appraisal/Failure COQ ratio 
Percent defect free 

Defect removal profiles 
Quality profile 

Quality profile index 
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TSP Weekly Tracking 


TSP teams track their status weekly using a defined 
process and the weekly status summary in the TSP 
Support tool. 


TSP Week Summary - Form WEEK 
Name 

Team 

Status for Week 
Week Date 


Weekly Data 
Schedule hours for this week 
Schedule hours this cycle to date 
Earned value for this week 
Earned value this cycle to date 
To-date hours for tasks completed 
To-date average hours per week 


Task Plan Task Earnedor Planned 
Tasks Completed or Due Resource Hrs. Actual Hrs. Plan Value Week 
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Earned Value Management 


TSP teams review progress at the weekly meeting using 
earned value tracking provided by the TSP support tool. 


100.0 


Percent Complete 
a 
i=] 
Qo 
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Resource Management 


TSP teams review resource utilization at the weekly 
meeting using analyses provided by the TSP support tool. 


1200.0 


—e— Cumulative 
Planned 
Hours 
1000.0 
—=— Cumulative 
Actual Hours 


800.0 


600.0 


Cun wiative Planned Hours 


400.0 


200.0 


0.0 
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Quality Management 


TSP teams use the Quality Profile as an early warning indicator 
of post-development defects. 


The quality profile uses five software quality benchmarks. 


Satisfied criteria are plotted at the outside edge of the chart. 


Component 2 Risk Factors Component 5 Risk Factors 


Inadequate 
design review 
a Time time results in Design/Code Time 

bo design defects we 
— Code Review Time escaping to Design Review Time oe 
test and 
production. 


Design Review Time ae ia Ss Code Review Time 


Unit Test D/KLOC Compile D/KLOC Unit Test D/KLOC* ™~ Compile D/KLOC 


High quality component Poor quality component 


Carnegie Mellon 


~“—==|- Software Engineering Institute 


Defect Removal Profile 


TSP teams use the Defect Removal Profile to track 
e plan and actual defects removed by phase 
e early vs. late defect removal plan 


Defects Removed by Phase 


Defects Removed by Phase for Assembly SYSTEM 


Phase 


—@e— Plan 
—a— Actual 
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Exercise Objectives 
The PSP Is the foundation for the TSP. 
This exercise provides 
¢ an understanding of the baseline process, PSPO 
¢ familiarity with the basic measurement forms used in 


the PSP 


Similar measures and forms are used in the TSP. 
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Basic Process Elements 


A process script and basic measures 
A project plan summary form 

A time recording log 

A defect reporting log 


A defect type standard 
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Basic Process Measures -1 


The reason to measure a process is to understand it. 
¢ how much time is spent in various activities 
e what is produced at various times 
e how many defects are injected and removed, and when 


With these data, engineers can better 
e plan and estimate the work to be done 
¢ evaluate the results 
e improve the process for the next project 
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Basic Process Measures -2 


TO measure the process, the work is divided into defined 
activities called phases. 


Each phase consists of 

e the task to be done during the phase 

e the entry criteria, or the items required before the work 
can start 

e the exit criteria, or the items that must be produced by 
the end of the phase 

e verification steps to ensure that the work Is properly 
done 
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Basic Process Measures -3 


The measures for each phase are 
e time spent (in minutes) in that phase 
¢ defects injected in that phase 
¢ defects removed in that phase 


The program size is also measured, but only during the 
postmortem phase at the end of the project. 


These measures provide the foundation for all PSP 
measurements, analyses, and planning. 
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Baseline Process Phases 


Baseline Process 


ow) 
© 
< 
oO 
O 
oO 
= 
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A Process Script 


PSPO Process Script 


Phase 
To guide you in developing module-level programs 


Entry Criteria ¢ Problem description 
PSPO Project Plan Summary form 
Time and Defect Recording Logs 
Defect Type Standard 
Stop watch (optional) 


1 Planning Produce or obtain a requirements statement. 
Estimate the required development time. 
Enter the plan data in the Project Plan Summary form. 
Complete the Time Recording Log. 


2 Development Design the program. 
Implement the design. 
Compile the program and fix and log all defects found. 
Test the program and fix and log all defects found. 
Complete the Time Recording Log. 
7 
time, defect, and size data. 


Exit Criteria ¢ A thoroughly tested program 
* Completed Project Plan Summary form with estimated 
and actual data 
Completed Defect and Time Recording Logs 
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PSPO Project Plan Summary 


The project plan summary holds project data in summary 


form. 


e planned and actual data 
¢ to date history 

e time in phase 

¢ defects injected 

e defects removed 


PSPO Project Plan Summary 


anning 


Date 
Program # 
Language 
Pla Act To Date To Date % 
Act To Dat To Date % 
Act To Dat To Date % 
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Time Recording Log 


Engineers use the time recording log to record 
e the time when they start on a project phase 
e the time when they stop work on a phase 
e the interruption time 
e the elapsed time less interruption time 
° comments 
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Defect Recording Log 


Engineers use the defect recording log to record 
information about all defects found in reviews, compiling, 
and test. 

¢ the defect number 

e the defect type 

e the phase in which it was injected 

e the phase in which it was removed 

e the time to find and fix the defect 

e a brief description of the defect 


lf the defect was injected while fixing a defect, that defect’s 
number is recorded. 
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Exercise Instructions -1 


Read through the PSPO process scripts (in the workbook) 
so that you understand the entry and exit criteria for each 
phase. 


Read JD’s scenario for program 1A and fill out the time 
log. The defect log and project plan summary are already 
filled out for you. 


Refer to the instructions for each form to determine what 
information goes in each field. 
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Exercise Instructions -2 


When did JD start? 
When did he finish? 
Was he interrupted? 


What process phase is 
this? 


Where should this 
information be recorded? 


JD begins work on assignment 1A 
[8:00] by reviewing the requirements in 
the assignment package, including the 
test requirements, to be sure he 
understands them. He copies the 
requirements to his note pad. Then, 
based on the data presented on past 
student performance and JD’s feeling 
about his own performance, he 


estimates this assignment will take 3 
hours and writes this on his note pad 
[8:06]. 
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Results 


How long did the project 
take? 


How many defects were 
removed? 


In what phase did JD 
spend the most time? 


What percent of JD’s time 
was spent in compile + 
fesi2 
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Exercise Summary 


The baseline personal process is simple and easy to use. 


The PSP forms simplify data collection and provide a 
convenient reference for planning future projects. 


The basic PSP time, size, and defect measures provide 
the data for the TSP. 


HOMEWORK: For systems engineering in your 
organization, how would the Plan Summary change? 
What phases of development would you define? 
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Building High-Performance Teams 


TSP builds high-performance teams from the bottom-up. 


Team 
SUT Cellars 


Teaming 
ST CHESS 
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2) 


Goal setting 

Role assignment 
Tailored team process 
Detailed balanced plans 


Process discipline 
Performance measures 
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TSP Structure and Flow -1 


In the TSP, each major project 
cycle or phase begins with a 
Launch. 


The Launch is a defined team 
planning process that also 
facilitates team-building. 


The team reaches a common 
understanding of the work and 
the approach. 


They produce a detailed plan to 
guide the next development 
phase or cycle. 


Carnegie Mellon 
Software Engineering Institute 


TSP Structure and Flow -2 


TSP has four principal > — 
development phases. —. 
e Requirements, High-Level 
Design, Implementation, > Postmortem 


Test (TSP default) 
¢ or a project-defined lifecycle 


TSP projects can start or end > cea 


on any phase. 
¢ from requirements through 


system fool Postmortem 
¢ requirements only > 
¢ high-level design only 
¢ as needed to do the work 


Postmortem 
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TSP Structure and Flow -3 


The TSP phases can and {> — 


should overlap. — 


The TSP development 
strategy encourages 
¢ incremental development 
e iterative development 
¢ multiple builds or cycles 
¢ work-ahead 


Postmortem 


Postmortem 


TSP permits whatever 
process structure makes the 
most business and technical 
sense to the team. 


Postmortem 


Postmortem 
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TSP Process Elements 


Checklists, specifications, 
standards, and other process 
assets (22), including 


TSP introduction sequence 


Launch planning guidance 
Executive tools such as checklists 
for planning assessment and 
quarterly reviews 


Forms (22), including 
— Time Recording Log 
— Defect Recording Log 
— Inspection Report 
— Process Inventory 
— Quality Summary 


TSP role specifications (12), 
including 


Meeting roles and responsibilities 
Inspection roles and responsibilities 
Customer interface manager role 
and responsibilities 

Process manager role and 
responsibilities 


Process Scripts (30), including 


— Overall development and 
enhancement process 


— Overall maintenance and 
enhancement process 


— Launch process 
— Test defect handling 
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The Launch Process Meetings 


Day 1 


1. Establish 
Product and 
Business 

Goals 


2. Assign Roles 
and Define 
Team Goals 


3. Produce 
Development 
Strategy 


Day 2 


4. Build Top- 
down and 
Next-Phase 
Plans 


5. Develop 
the Quality 
Plan 


6. Build Bottom- 
up and 
Consolidated 
Plans 


Day 3 


7. Conduct 
Risk 
Assessment 


8. Prepare 
Management 
Briefing and 
Launch Report 


Day 4 


9. Hold 
Management 
Review 


PM. Launch 
Postmortem 
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The TSP Launch Artifacts 


Business needs 
Management goals 
Product requirements 


How What 


Team goals Team Task hour Teamroles Quality plan Risk 
strategy plan evaluation 
Conceptual Task plans 
design Team Schedule Alternative 
process plan Detailed plans 
Planned plans 
products Earned- 
value plan 
Size 


estimates 
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TSP Project Tracking -1 


Project tracking in the TSP Is based on the principles and 
measures used in the PSP. 


The detailed team and individual plans facilitate precise 
project tracking. 


Each team member is responsible for 

e gathering data on their work 

e tracking status against their personal plan 
e keeping the team informed 

e the quality of the work they produce 
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TSP Weekly Meeting 


Manager’s report (team leader) 

e new issues and developments 

Role reports (8, more or less) 

¢ customer/requirements, design, implementation, test, 

planning, process, quality, support 

Risk report 

e status and changes in assigned risks 

e impending flag dates and required actions 
Project status 

e individual and team (planning manager) 
Next week’s plans 

¢ individual tasks 

° dependencies (e.g. reviews needed) 

e task, hour, EV goals 
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TSP Project Tracking -2 


Project tracking in TSP is based on 
e the team’s plan 
e task hour and task completion data 
e plan and earned value 


Individual plans facilitate precise project tracking. 


Team members are each responsible for 
e gathering data on their work 
e tracking status against their personal plans 
e the quality of the work that they produce 
¢ keeping the team informed of their progress 


Individual team member data are consolidated each week 
so that the team can assess progress against goals. 
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The WEEK Summary 


The weekly team meeting is the forum that the team uses to 
e track progress against the plan 
e track the status on the project's issues and risks 
¢ communicate with each other 


TSP Week Summary - EK 
Name Con eam Plan Date 

Team Sec q 
Status for Week 
Week Date 


Plan/ 
Actual Actual 


Cycle 


Weekly Data Plan 
Project hours for this week 80.0. 
Project hours this cycle to date 
Earned value for this week 
Earned value this cycle to date 
To-date hours for tasks completed 


Plan Actual Earned Planned Plan Hrs./ 
Hours Value Week Actual Hrs. 


Assembly 
SYSTE) 


Hours 
14.0 


Resource 
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Maintaining the Team’s Schedule 


The team manages its commitments by using the data it 
collects. 


The team determines how it is doing against its plan. 
If the team is falling behind, it determines 
¢ what is the likely cause 


e what the team can do to maintain its commitment 


The team informs management if the commitment cannot 
be maintained or if management help is needed. 
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Determining Status Against Plan -1 


Two things are important here. 
e the team’s current project status 
e the team’s projected completion date 


Current status is determined using data on the WEEK 
form. 


TSP Week Summary - Form WEEK 
Name Consolidated Team Plan 
Team SecuritySystem Upgrade 
Status for Week ( 5a 


Week Date _ 1/31, 


Date 2/7/2000 


Cycle 


Weekly Data Plan Actual Actual 
Project hours for this week 


(plan EV todate — actual EV todate) 
(actual EV todate / current week ) 


weeks behind = 


| 
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Determining Status Against Plan -2 


Projected completion date can be determined using data 
on the WEEK form and the original planned weeks. 


TSP Week Summary - Form WEEK 
Name Consolidated Team Plan 
Team Milleniwm—Upograde 
Status for Week 
Week Date _ 1/31/2000 


Date 2/7/2000 


Plan/ 
Weekly Data Plan Actual Actual 
Project hours for this week 80.0 1.16 
Project hours this cycle to date 400.0 
arned value for this week 10.3 
Earned value-this cycle to date 40.2 


To-date hours for tasks-co 


(100— actual 


k = 
weeksto go (actual EV todate)/(currentweek) 


weeks behind 


original 
= (weeks to go+current week )— 
at completion 


planned weeks 
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Identifying Estimating Problems 


The cost performance index (CPI) shows how the team is 
performing with respect to the effort estimates in the plan. 


plan hours for completed tasks 
actual hours for completed tasks 


CPI = 


The CPI is available on the WEEK form. 


TSP Week Summary - Form WEEK 
Name Consolidated Team Plan 
Team Security System Upgrade 
Status for Week 5 . 
Week Date _ 1/31/2000 


Date 2/7/2000 


Weekly Data 
Project hours for this week 
Project hours this cycle to date 
Earned value for this week 


Earned value this cycle to date 40.2 30.0 34 
To-date hours for tasks completed 293.0 303.8 C096 ) 


Carnegie Mellon 
Software Engineering Institute 


Interpreting the CPI 


A CPI of 1 means 


What does this imply about the accuracy of the individual 
estimates? 


Assuming the team is achieving the planned task hours, 
what does this imply about schedule performance? 


What does a CPI of 0.5 imply about 
e effort estimates? 
e schedule performance (assuming the team is achieving 
the planned task hours)? 
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Interpreting the CPI (continued) 


What does a CPI of 2 imply about 
¢ effort estimates? 
e schedule performance (assuming that the team is 
achieving the planned task hours)? 


What general characterization can be made about 
schedule performance based on the CPI? 


Schedule growth (due to effort estimates) = 1/CPI 


Projected schedule = Original plan weeks/CPI 
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Interpreting Task Hour Data 


The task hour data is in the form WEEK and can be 
interpreted similar to the effort for completed tasks data. 


*Status for Week 
eWeek D 


*Plan/ 
¢Weekly Data Al 
Project hours for this week 
Project hours this cycle to date 
eEarned value for this week 
eEarned value this cycle to date 
¢To-date hours for tasks completed 


lf (Plan hours to date)/(Actual hours to date) = 2 
e What does it mean? 
e What is the effect on schedule performance? 
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Interpreting Task Hour Data continued) 


lf (Plan hours to date)/(Actual hours to date) = 0.5 
e What does it mean? 
e What Is the effect on schedule performance? 


What general characterization can be made about 
schedule performance based on the plan/actual task 
hours? 


Schedule growth (due to task hours) = plan/actual 


Projected schedule = Original plan weeks * (plan/actual) 
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Improving Task Hours 


Average task hours per developer per week were improved from 
9.6 hours to 15.1 hours through quiet time, process documentation, 
more efficient meetings, etc. 


Average Task Hours Per Week 


Task Hours 


4 —¢— Avg. Task Hours - Week 


—m— Avg. Task Hours - Phase 


Source: Allied Signal 
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What is Quality? 
Basic definition: Meeting the user’s needs 


There are three categories of product quality. 

e functionality 

° properties (e.g., safety, security, privacy, usability) 
¢ defects 


A software-intensive product can’t be safe or secure until it 
is nearly defect-free. 


Most current software-intensive processes are 
preoccupied with removing defects. 


Little or no time is left for the other aspects of quality. 


Carnegie Mellon 
Software Engineering Institute 


The System Quality Problem 


Software quality problems are largely caused by defects. 
e Defects are injected by the product’s developers. 
e Even experienced and capable developers inject many 
defects. 
e Each defect is a potential system failure. 
e A significant fraction of software defects can be avoided 
or mitigated by effective systems engineering. 


Current practices often rely on testing to remove these 
defects. 


Testing is necessary but, for finding and fixing defects, it is 
¢ time-consuming 
° expensive 
¢ ineffective 
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The Defect Problem 


Programs are complex products. 
e Small programs have thousands of instructions. 
e Large programs have millions of instructions. 
e These instructions are individually produced. 
e Each instruction must be precisely correct, beginning 
with the problem statement. 


Software effort has a multiplying effort on systems 
engineering defects. 


On average, even experienced programmers inject a 
defect about every 10-to-12 instructions. 
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Testing 


A single test 
e exercises the product under one set of conditions 
¢ produces correct or incorrect results 


If there is a problem, developers must find the defect, fix it, 
and then test the fix. 


For products with many possible operating conditions, 
many tests are required. How many of these tests are 
defective? 


Projects that rely on testing for quality spend a lot of time 
and money on testing. 
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Testing Takes a Long Time 


Magellan Spacecraft — 22,000 LOC 


Cumulative Defects 


200 
180 
160 
140 
120 
100 

80 


—— All 
~~ Non-Crit 
Critical 
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Testing Effectiveness 


Large complex systems cannot be exhaustively tested. 
e It is impossible to test every operating condition. 
e Testing must focus on only the most frequent 
conditions. 
e Extensive user testing finds even more defects. 


Testing finds a percentage of the defects in a product, 
usually less than 50%. 


To get a quality product out of test, you must puta 
quality product into test. 
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Testing is Ineffective 


Overload 


Hardware 
Configuration failure 


Safe and secure 
region = tested 
(shaded) 

Unsafe and insecure 
region = untested 
(unshaded) 


Operator 
error 


Resource 
contention 


Data error 
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Reviews and Inspections Save Time 


10000 


System Test is the least 
1000 | efficient phase in which to 
remove defects 


100 | 
25 


Time in Minutes 


Design Design Code Code Unit System 
Review Inspect. Review Inspect Test Test 


Source: Xerox Defect-removal Phase 
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Why TSP is Faster and Better 


With TSP 
¢ most defects are removed by reviews and inspections 
e few defects are left for testing 
e testing takes relatively little time 


By using TSP, organizations can 

¢ cut testing times by 80% or more 
e shorten schedules 

e reduce costs 

¢ produce better products 


Testing should verify that the development process 
worked well, rather than fix its exported problems. 
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Measuring Quality 


TO produce quality systems, the quality of all its parts must 
be measured and managed. 


These measures must be made at every step in the 
process. 


With TSP and the underlying PSP principles, developers 
use quality measures to manage the quality of their work. 
The developers 

e inject fewer defects 

e remove most defects soon after injecting them 
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TSP Quality Measures 


There are many potential quality measures. 


With the TSP, every product element and every process step 
can be measured. 


Quality Measure 


Total defect density ———— number of defects ounuE in development, per 
unit of size 
Compile defect The number of defects found in compile, per unit 
densit of size 
Product 
Quality | Test defect density ips number of defects found in test, per unit of 
The percent of system modules or components 
HOOT ISOE TES that had no defects in a defect removal phase 


The percent of defects in a product that are found 
during the phase 


; The volume of code or design that is reviewed per 
Review rate 
Process Jel 
: Defect removal rate The hourly rate at which defects are removed in 
Quality 
- defects/hour reviews or inspections 


Quality profile Composite picture of a module's process qualit 


Process quality index |A composite value representing the five quality 
(PQI) profile dimensions 


Phase yield 
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Quality Implications 


With proper training, guidance, and motivation, most 
developers can produce near-defect-free programs. 


Does the same hold true for systems engineers? 


With essentially defect-free products 
¢ testing times are sharply reduced 
¢ delivered products work 
¢ maintenance costs are reduced 


The key is the engineer’s ability to produce defect-free 
products. 

¢ measure quality 

¢ manage quality 

¢ personal quality commitment 
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Quality Goals and Plans 


With data, TSP teams can 
¢ set measurable quality goals 
e make quality plans to meet these goals 
e estimate the defects injected and removed in each 
phase 
¢ track the work to see if they are meeting their quality 


plans 
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The TSP Defect Model 


At each step of development, defects are injected, 
removed, or possibly both. 


For each step: 
Defects Out = Defects In + Defects Injected — 
Defects Removed 


Defects In = Defects Out from the previous step 
Defects Injected = function of time in production activities 


Defects Removed = percentage (usually much less than 
100%) of Defects In + Defects Injected 
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Example: Planning for Quality -1 


A TSP team plans to develop 20 KLOC. 


The goal is a design review yield of at least 70%. 
¢ The plan shows 442 hours in detailed design. 
¢ Data show that developers inject 1.3 defects per hour in 
detailed design. 
¢ Data show that they remove 3 defects per hour in 
detailed design reviews. 


What is the minimum design review time required to 
remove these defects? 
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Example: Planning for Quality -2 


Defects injected 
e 442 hours of design 
e 1.3 defects injected per hour 
e 1.3*442 = 574.6 defects injected 


Defect removal 

¢ 574.6 defects total 

e 3 defects removed per hour 

¢ 574.6/3 = 191.5 hours of design review time 


The team should plan on 191.5 hours of review time. 


To achieve a 70% yield, they must spend at least 
0.7*191.5 = 134.1 hours in design reviews. 
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Example: Planning for Quality -3 


Assume that 
e no design reviews are done 
¢ 4 of the design defects (.5 * 574 = 287) can be found 
by integration testing at 5 hours/defect 
e % of the remaining defects (I.e. 4% of % or .5 * 287 = 
144) can be found in system testing at 10 hours/defect 


How much time will integration and system testing take? 
How much time will be saved by doing design reviews? 


How many design defects will likely remain for your 
customers to find? 
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Maintain Process Discipline 


TO produce quality systems, every part must be of high 
quality. 


This is possible only if every developer consistently 
follows a quality process. 


To consistently follow a quality process, each member of 
the development team must 
e be properly trained (with the PSP or equivalent) 
¢ work on a disciplined team (with the TSP or equivalent) 
e have coaching support and management guidance 
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Management Support 


People do not naturally do disciplined work. 


To ensure disciplined work, management must 
¢ train and support the developers 
e ensure that the developers’ work is guided and 
monitored 
¢ provide coaching assistance 


Management must also 
e build and maintain effective teams 
e ensure that all team members are trained and willing to 
follow the process 
e recognize and reward quality work 
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TSP Quality Messages 


High-quality processes produce high-quality products. 


Quality work is not done by accident; it requires discipline, 
commitment, management, and measurement. 


Quality work saves time and money. 


The cornerstone of a high-quality software process is early 
defect removal. 


TSP shows teams how to efficiently remove defects at the 
earliest possible point in the process. 


Carnegie Mellon 
Software Engineering Institute 


Your Organization is Unique... 


...ut most organizations share common problems. 


An organization can change under duress, or it can 
change in response to leadership. 


Duress can lead to undesirable consequences since, by 
definition, it is trying to get away from whatever is causing 
the duress. 


Only leadership can take an organization reliably ina 
desired direction. 


Where will you lead your organization? 
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Thank you! 


Contact information: |dm@sei.cmu.edu 


Contact a PSP or TSP transition partner: 
http://www.sei.cmu.edu/collaborating/partners/trans.part.psp.html 


Contact SEI customer relations: 
Software Engineering Institute 
Carnegie Mellon University 
Pittsburgh, PA 15213-3890 
Phone, voice mail, and on-demand FAX: 412/268-5800 
E-mail: customer-relations@sei.cmu.edu 
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Trademarks and Service Marks 


The following are service marks of Carnegie Mellon University. 
¢ Team Software ProcessS™ 
¢ TSPSM 
¢ Personal Software ProcessS™ 
¢ PSPS 


The following are registered in the U.S. Patent & Trademark Office by 
Carnegie Mellon University. 

¢ Capability Maturity Model® 

¢ CMM® 

¢ CMMI® 


Raytheon 


“Who needs that process stuff anyway?” 


A practical example of process discipline to improve deployment 


Raytheon Missile Systems 
William Borkowski, J r. 
November 14, 2006 
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Rayilheen 
Author Contact Information 


¢ William Borkowski, Jr 

e Engineering Fellow 

° Certified Raytheon Six Sigma Expert 
e Raytheon Missile Systems 

e Tucson, AZ 


¢ wjborkowski@raytheon.com 
e 520.794.2444 
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Raytheon 
Organization and Accomplishments 


Raytheon Missile Systems, Headquarters Tucson, AZ 


Employees: 11,000 
2005 Sales: $4.1B 


World Largest Appraised SEI 
CMMI Level 3 Organization 
December 2004 


SW-CMM® Level 5 in 
November 2001 


SE / SW CMMI® V1.1 Staged 
Representation Level 3 in 
December 2004 


Me MM HH 
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Raytheon 
The Situation 


¢ “The Conflict” 
° No insight into progress toward CMMI level 3 goal 


¢ What Specific & Generic Practices by PA assessment ready across Front 
Runners 


¢ “The Questions” 

How much evidence do we need? 

How do we know where we are? 

How many process areas are deployed? 
What evidence are we going to collect? 

Where are we with evidence collection? 
How much do we collected? 

Where is it? 

Etc, Etc, Etc 


e “What to do”? 


— R6S Project - 
° Show 6 stages - Visualize, Commit, Prioritization, Characterization, Improve, Achieve 
° Tools - Brainstorming, affinitizing, prioritizing, root cause analysis, & consensus building 
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Key elements of our approach 


GOAL: 
¢Use a proven approach to rapidly fix a major problem 


¢Provide quick, accurate, detailed graphic results 
to improved visibility into SCAMPI readiness 
eQuick response < 6 weeks 
° Vision 
— Create a vehicle that can be used to track progress of the 
RMS CMMI Project toward attaining level 3 
e Requirements 
— Involve key stakeholders 
° Leadership 
° Deployment teams 
© Process Area Experts 


Raytheon 


— Collecting & reporting status to the stakeholders in a consistent, 


deterministic, and timely manner 
— Graphic reports of key metrics 
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Raytheon 
Visualize 


¢ Problem Statement 


¢ The CMMI Deployment Tracking Spreadsheet & Reporting Tools 
where difficult to use, inaccurately populated, and consequently not 
readily usable by the CMMI Leadership Team to monitor project 
status 


° Vision 

¢ Implement a deployment tracking process to monitor progress 
towards attaining CMMI Level 3 at RMS 
— Reports tracking progress are required 


e Scope 
¢ Deployment Tracking Spreadsheet updates shall be ready in less 
than 6 weeks 


© 2000, Raytheon Company. All Rights Reserved. 
Revised: September, 2002 - 20446AGP 1-6 


Raytheon 
R60 Process Model 


“Imagine the Future” 


“Celebrate Vg — YY 


‘ ' . 
men Commit to 
A Cc sts = ts 4 Achieve MEO sisal C ommit Cc h a n g e iT] 
Tomorrow” 


“Design and “Determine 


Implement Improvement 


Improvements” XK wa Priorities” 


(@itinlacseys- 


“Define Existing Process 
and Plan Improvements” 
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Raytheon 
Applying R6o0to What We Do 


Performance 
Gap — Pareto 


Analysis 


Goal Projected 


Categories 


Achieve & Improve “Big-Hitters” 


Root Cause 
Improvement Analysis 
Plan 
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Commit Raytheon 


¢ Met with Sponsors 
— Outlines approach & Project requirements 
¢ Obtained commitment to proceed: 
® Vision, Commitment (do whatever it takes), Scheduled this as an 
agenda item for Project all hands meeting (5/18/04) to discuss & 
gather survey comments 
° Meeting agenda with stakeholders 
» Project Vision & Deliverables 
» Project Assumptions 
» Data Gathering approach 
» Determination of current Process Flow 
» Deliverable 
» Assumptions 
» Stakeholder Tasks — how they could make this successful 


M 
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Deliverables Raytheon 


e Updated Deployment Tracking Spreadsheet 
e Updated Run Rules & Process Flow & training 


° Reporting Charts 
— Deployment & Evidence Rate Charts (1 per Program & Roll up) 
— Program Plans / TDs by PA and Practice (1 per Program) 
— Program Opportunities by Program & PA 
— Evidence Status by Program & PA 
— Program Evidence by PA & SP (1 per Program) 
— Program Evidence by PA & GP (1 per Program) 
— Rollup of all evidence across all Front runners required for SCAMPI 


e Demonstrated accurate information usable by the CMMI Project Team 
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Prioritization Raytheon 


¢ Devised plan to survey PAEs, MOR’s, POR’s, & Deployment Leads 
— Got stakeholders opinion 
¢ Fashion a solution that met Sponsor’s vision 
— Usable by different class of users 
° Deployment teams 
° Process area Representatives 
® Leadership team for status 
° Appraisal team for SCAMPI 


— Reporting requirement 
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Raytheon 
Requirements Context Diagram 


Appraisers 
Deployment Team use data for 
& PAEs Provide data Appraisal 


Hanson SPC Tool 
Tool ¢ Status of 


Tracking Spread Sheet 
* Direct & Indirect * Evidence SP & GP 
evidence identified * Context Comments Evidence 


¢ Execution opportunities 
¢ Program POC 

¢ Plans Updated 

¢ Process Training 

¢ Evidence to Hanson tool 


° Evidence Reviewed 


Observations 
*Fully 
-Largely 
*Partially 
«Weakness 
«Notes 


Status Reports 
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Shared Activities (OLD) Process Flow ®#vtheon 


DT Posts Program 


Generate 
Reports 
Master Deployment PAEs Checks out 


i DT /PAE color Return 
Tracking Spreadsheet Deployment De lavnient 
in Docushare @ Tracking Codes Deployment ploy’ 


CMMI 2004 Admin Tracking Spreadsheet Tracking 


Spreadsheet Status Spreadsheet 
i agian to Docushare 


Metrics 


External 
Evidence 
Reviews 


@ 
CMMI 2004 
Admin 


IRT does 
Es Program Internal 
Populate CMMI Metrics evidence 
Deployment Reviews 
Tracking 
Spreadsheet 
Evidence: DT /PAE 
Evidence * Direct Populates 
opportunities * Indirect URLs into 
Identified * Execution Ops. Hanson 


& Captured * Prog POCs Tool 
Review 


* Plans 
Program PA * Training 
Training 
Identified 
Plans Created Evidence 
&/ or Updated : (Shoulder to 
DT Monitors Shoulder 


Spreadsheet to & Mini 
capture execution PBAs ) 


Pre — Meeting Kickoff Mtg 


Deployment 
& PAEs DTs & PAEs 


opportunities 
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Characterization - 1 of 2 


¢ Surve 


Raytheon 


uestions 


— Describe your Current State 

— What are your top 3 Desirables about the Current State? 

— What are your top 3 Undesirables about the Current State? 
— What specific Run Rules have you developed for your team? 
— What is your Ideal Future State? (3 key descriptions) 

— What are your change recommendations? Be Specific 

— What data do you find useful to you? 

— What data is not useful to you? 


¢ Received 182 inputs from 5/18/04 data gathering session 


72% (122) are covered by proposed solution 
28% (47) are not covered 
(13) Use one tool 
» The Spreadsheet and the Hanson tool both have unique capabilities 
(9) Opinions about what is useful data 
» Varies greatly from nothing is useful to everything is useful 
(6) Spreadsheet control 
» Decision is to not lock spreadsheets and to allow docushare checkout 
(4) Consistency between Spreadsheet and Hanson tool 
» Required to ensure accuracy of content and eliminate confusion on 
what is loaded versus what is required 
13 inputs are non-actionable 
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Characterization - 2 of 2 Raytheon 


¢ Deliverables — See Tasks Below 
¢ Improvement Plan Schedule 


Task Owner Start End When Total L 


0 
io 
9 
0 
, 
0 
0 
m 


Develop proposed task list 5/14/2004 _|5/16/2004 


Create proposed Problem Statement i 5/14/2004 5/17/2004 
Create proposed Vision i 5/15/2004 _|[5/17/2004 
Create proposed expectations 5/16/2004 [5/17/2004 


AM 

AM 

AM 
Create proposed deliverables Bill 5/17/2004 5/17/2004 AM 
guidelines 
Gather/Understand all Reporting Charts Mike 5/17/2004 |[5/17/2004 AM 
Identify & Invite Stakeholders to 5/18 PAE/DT Meeting Mike 5/17/2004 5/17/2004 PM 
Develop questions to ask Stakeholders at 5/18 PAE/DT Bill 
Meeting 

PM 


Prepare agenda & Presentation for 5/18 PAE/DT Meeting Mike / Bill 5/16/2004 |5/17/2004 


RIB] NIN} O;OJOJOJOIN 


: ili I] 


N 


Gather data from all Stakeholders (may Nave to call some |Mike / Bill 5/18/2004 5/18/2004 AM 
directly) 

Develop Current State Descriptors 5/18/2004 _|[5/18/2004 
Develop a Current State Process Flow 5/18/2004 __|[5/18/2004 


PM 

PM 

Create Future State Descriptors from Stakeholder inputs Mike 5/18/2004 5/18/2004 PM 
Determine Root Causes for Undesirables Mike / Bill 5/19/2004 5/19/2004 M 
M 

M 

M 


ia 


i A 
Identify the Gaps between Current & Future States Mike / Bill 5/19/2004 _|[5/19/2004 A 
Merge Root Causes & Gaps & Affinitize the list Mike 5/19/2004 5/19/2004 Lunchtime 


Brainstorm Solutions to the Affinitized List Mike / Bill 5/19/2004 5/19/2004 
Prioritize Brainstormed Solutions Mike / Bill 5/19/2004 [5/19/2004 
Create a Business Case (Desccription, Benefit Mike / Bill 5/19/2004 5/20/2004 
Implementation Process & Cost, etc) for Key Brainstormed 

Solutions 

Measure proposed Solutions against Future State Mike 5/20/2004 5/20/2004 
Descriptors 

Weekly Progress Report Due Bill 5/20/2004 5/20/2004 
PM 
Mike-/ Bill |s/28/2004 [5/25/2004 
Train Stakeholders Mike / Bill 5/25/2004 _|[5/25/2004 
[Weekly Progress Report Due BN 5 /2772004_ [5/27/2004 _ [PM 


Stakeholders Update Spreadsheets Stakeholders 5/25/2004 _|6/1/2004 
Correct Spreadsheet Errors Mike / Bill 6/1/2004 6/4/2004 


P 
P 


QIN = WIW)W) WWW 


Alp) N 
ojo] o 


R/O|R 
o|o| 
fo) 


Aly 
| 
oly 


N 
fe) 


Measure Success to Deliverables Mike / Bill 6/4/2004 6/4/2004 
Complete Project Submittal Mike / Bill 6/A/2004 6/4/2004 
Celebrate Mike / Bill 6/4/2004 6/4/2004 PM Infinity 


N 
fe) 
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Improve Raytheon 


Executed plan 


¢ Suggested improvements to meet requirement as project 
progressed 


° Got expert & necessary help to transition current 
state matrix to improved form 


Trained stakeholders 


¢ Using the matrix to track progress 


Using output reports to show status & to focus team efforts 
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Raytheon 


New Process Flow 


fcccccccccccco 


PAE & DTs PAE / DT 
Establish |= >| Hold Kickoff Mtg 
With Program 


Engagement 
Run 


PAE /DT/Program 
Discuss Process 
Area 
Requirements 


PAE 
PAE/DT/Program IDs primary 
Identify Plan plan or TDs & 
creation & populates 
update spreadsheet 


: Assessors 
requirements when Plans are 
comments 
complete 


PAE uploaded to 
SPC tool 


CMMI team 


does Process 
Area 
Training 


Hanson 
Tool 
inputs 
uploaded 
to SPC 


PAE /DT/Program 
Identify 

Unique Program 
Requirements 


(Deployment Activities }______ 
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racking Spreadsheet & Hanson Tool Activities 


1-17 


Hanson Generates 
observation reports 
& communicates to 


Status 
Reports 
Generated 


Raytheon 
Tracking Run Rules 


= =] 
cial Run Rule 
1 Each Practice willhave a minimum oF 1 direct evidence item and 1indirect evidence item ? affirmation 
= 1Evidence item perrow 
BJ Deployment Tracking Spreadsheet column owners are ultimately responsible For the completion of their columns 
4 Deployment Tracking Spreadsheets are available bo all Ch) team members Folr docushare checkout 
When pou update 4a Oeployment Tracking Spreadsheet, moriky applicable OTS & POEs. OTS will verify link= are ok and PSS will werify combenk i= 
= ok 
= when adding of modifying evidence in the Hanzon bool, add “Mew in Front oof fhe document name 
When reviewing andfor adding cantexst comments for evidence that has been placedin the Hanson tool, remove the “MEW” From the Front oF the 
¥ document ore 
Each Program OT will maintain a golden Deployment Tracking Spreadsheet in +s separate location a= insurance in the euent of 4 docushare data 
= crash 
= Consistency between the Deployment Tracking Spreadsheet and the Hanson bool will be maintained 
10 Seoring for a column is nor “Complete” until all activity on the Evidence rows is Finished 
i Foo evidence is available type inUhlo Opportunity” in the “Load Latest Review Fesulke" column 
a IF Ro Opportunity” if used, reduce the "Total # of Opportunities” count at the bothom of the Deployment Tracking Spreadsheet 
1s Maintain each program's Deployment Tracking Spreadsheet in the docushare admin area For each program 
14 The Practices row order onthe Oeployment Tracking Spreadsheets will mot be modified 
15: The Practices row is For “scoring” only. 
16 Evidence rows are used to gather evidence information 
ra “Tiohal # of Opportunities” i= defined as the # of SPs and GPs, minus any identified a= “Blo Opportunity” 
1h Foor the Flan column to be considered “Complete”, the Plan must be identified, created f updated, and be ready For program use 
19 IFoo Flan if applicable For a Practice, boad the applicable (IPOSMRMIS Task Descriptor 
270 The Flan is loaded at the Practice level 
a1 IF additional Plans need to be referenced, place them inthe “lomments”" oobumn 
Foor the “Identify & Complete Execution Opportunity” column, Follow the dabe (2004/0418) - opportunity (Tam of Three] - Program POC [name] 
a SON nnn. (pou donk Boy cee Specikic day, use the lace day oF the month. 
The “Identify Execution Opportunity” column is considered “Complete” when the opportunity information For each evidence row i= captured oan 
a the Deployment Tracking Spreadsheet. 
24 To be able to “Load Evidence”, the Execution Opportunity must hawe have occurred and the information captured 
2h When Review resulke are anything bur “Fully, the Practice is considered red and the “Complete” for all columns if removed. 
Selece “Loaded in Hanson Tool" to indicate Contest Comments have been loaded into the Hanson Tool Selec “hlone” if mo Contest 
-e Comments are required. Selecr “Blank.” if Context Comments have mot pet been loaded into the Hanson Toc. 
ar IF a Practice has mo indirect evidence available, type] “AFFirmation Required” in the “Comments” column onthe Practice now 
When a review results in & “Largely”, “Partially, or “hor, a “Red” cobor code is reported. OT removes the “Complete” from columns FG, and 
H. Once a “Complete” i= reloaded (which means that an update has occured) into one of the columns C.F, oH, the OT changes the review 
ud : 4 resulk= column Heo “Blank”. This resets the colors to restart progressing through yellow and green. 
when uploading URLS, donot paste the document name, because if the document is changed, the link willbe broken. Example bo use: 
httpeids.cms.ray.comidscgilds.pylGet!File-209214 . Awoid: httpids.nms_ray.comidscgilds.pytGetlFile-20392147T TL_Master_Status_ Template .uls 
it: | . Browse bo the File using the DocuShare client then right-click on the File, edit, copy UFL. 
IF column E (ldentify Execution Opportunity] has not been completed on an evidence row, and the opportunity has already occurred and been 
loaded into the Hanson Tool then an albernative 00 loading “Oate - Oppeortunity - PO" i= to load “OBE - FP On”. Leaving the cell “Blank” or using 
ao “na is mot a suitable alternative. 
ES | Order evidence supporting each practice by importance in the Hanson Tool. 
To assist the 4s2essors review, Context Comments should cantain the Following 4 parts, a= appropriate: 
1 Spplicable page # oor location in the evidence document; 
2. ‘what part oF the practice of sub-practice the evidence answers; 
3. Which thread the evidence ties bo; and 
ba 4. Historical back ground!perspective of the document 


© 2000, Raytheon Company. All Rights Reserved. 
Revised: September, 2002 - 20446AGP 1-18 


Deployment Tracking Spreadsheet Definitidif¥""°™ 


Practices Rows 


Evidence Rows 


Opportunity Scoring 


Columns on this row are for “scoring” only. Words placed in the 
cell relate to color codes. 


For Column B, type in the applicable Plan or IPDS@RMS Task 
Descriptor 

For Columns C, E, F & G on the SP row, typing “Complete” into 
the cell indicates task completion 


For Column H, type in the scoring result received from the latest 
review 


Evidence and all actions associated with Columns C —- G are 
captured in the applicable column 


At the bottom of each worksheet 

Total # of Opportunities = the total # of SPs & GPs. This should be 
a fixed # 

(Exception) If “No Opportunity” is identified for a specific practice, 
then remove that SP from the count 

Total # Complete=the # of Practices Rows that 

1. Have aPlan or TD loaded 

2. Have “Complete” loaded 

3: Have a Latest Review Result loaded 


In SP 1.1 Column C (Identify Evidence), 
when all applicable evidence is loaded into 
the deployment tracking spreadsheet, type 
“Complete” in the SP row Column C cell 


In SP 1.1 Column C (Identify Evidence), type 
in evidence “xyz” 


Column C (Identify Evidence) has 

Total # of Opportunities = 20 because there 
are 21 SPs & GPs on the worksheet minus 1 
no opportunity 

Total # Complete = 6 because “Complete” 
was loaded on the SP Row cell 6 times 


Shaded areas No input is needed in the cell Leave blank 
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Raytheon 
Data Gathering Analysis 


Reporting Color Coding Process Flow 


Blank 


*Evidence gathering *Columns B-E *Columns F-G *Column H is 
just beginning completed completed “Fully” 
Red changes to Yellow 


or Green once action 
has been taken and the 


spreadsheet updated 
¢Column H is “No *Column H is 
Opportunity” (no anything other than 
opportunity to “Fully”, “No 
gather evidence for opportunity” or 
this PA Practice Blank 


exists 


A = 
PAE- 
SANTILLAN / 


Program: NFIRE 
MARSH 


Identify as 
Direct or 
Indirect 


(PAE) 


Practice Comments 


‘Complete 
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Updated Deployment Tracking Spreadshe@@¥™™°0" 


Program: NFIRE 


Example 


Process Area: TS PAE: Hector Esparza DT: Bill Borkowski / Mike Ard 


Identify & Gather Evidence 


Identify Identify Identify Identify 
Primary Plan | Evidence (PAE) |** Pa Execution 
(that is pee nialivesct Opportunity 
c program use) Or {Date [yyyy/mm/dd] - 
Practice Task (PAE) | “opportunity - POC} Comments 
Descriptor (is (PAE) 
identified) 
(PAE) 
M&A Plan Complete Complete Complete Complete 
2004/04/18 - Team of 
AZIE TAVINSOM. LOM 
xyZ D Three - Bhalala xyz @raytheon.com None 
" i 2004/04/11 - Worksheet i ne 3 
rs Markups - Bhalala rst@raytheon.com page 
M&A Plan 
Complete Complete Complete Conplete 
F 2004/04/18 - Team of 
pid 6 I SIE) 
ghi D Tiree = BAIA xyz@raytheon.com None 
ZOOMOAITT = Worksheet rst@raytheon.com Table 2 


SP 2.2 Establish and maintain a technical data 


Markups - Bhalala 


package. 


Complete 


2004/04/18 - Team of 
Three - Bhalala 


SP 2.3 Design comprehensive product-component 
interfaces in terms of established and maintained 
criteria. 


SP 2.4 Evaluate whether the product components 


should be developed, purchased, or reused based 
on established criteria. 


Total # of Opportunities 20 
Total # Complete SD) 
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2004/04/11 - Worksheet 
Markups - Bhalala 


Bn. ee ee 


ee eT 
20 20 20 
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Tracking Matrix Output #1 Raytheon 
Evidence by PA 


Activities Complete Status (TTL){old Rate Charts} @ Total # Complete 
B Total # of Opportunities 


140 


120 


100 


Inchstones 


CM DAR IPM MA Pl PMC PP PPQA RD RSKM REQM SAM TS VAL VER 
Process Area 
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Tracking Matrix Output #2 Raytheon 
Overdue Report 


CMMI Evidence Overdue Report by Program 


pit ProgA | CMT SP41 [2005/06/24 Mary Process___|ENB Table of Contents 
}2{ ProgA | CMT SP 144 | 2005/06/30 John Practice _—([SpecificationTree 
}3]) ProgA | CMT SP 141 | 2008/06/30 John Practice |[BoostFlight DVT DrawingTree 
}4{ ProgA | CMT SP 144 | 2005/06/21John Practice _—[DrawingNumberList. 
|S] ProgA [| CMT SP12 | 2005/06/22 Mary Process [DateAccessionList(DAL) 
}6{ ProgsA | CMT SP12 | 2005/06/21 John Practice | PDM control procedure for work authorzationforCMOP 
[7{  ProgA {| cM | P12 | 2008/06/22 Mary Process |E-mailforENB Structure 
[8] ProgA | CMT SP 13) | 2008/06/30 John Practice [Functional BaselineRepot(BFDVT) 
}9{ ~ProgA | CMT SP 143 | 2005/06/30 John Practice _—«([SOW DescribingFunctional Baseline, 
10] ProgA [| CMT SP13 | 2005/06/24 Joe Procedure [SoftwareBaseline 
a 
| ProgA | CMT SP143 | 2005/06/24 Joe Procedure _—|Software Synergy/PDM Baseline View 
So | a 
[14] ProgA | CMT SP 13 | 2005/06/24 Joe Procedure _—[E-Mail SchedulingSoftwareCCBAgenda 
15] ProgA [| CMT SP13 | 2008/06/24 Jane Audit [Hardware CCB agendaand minutes: DatedTBD 
fio] ProgA | CMT SP27 [2005/09/01 John Practice [Change Notification(TBD) 
[t7{|_ProgA | CM] P21 | 2005/06/24 Jane Audit _—iinitialReleaseMemo (IRM) 
fis] ProgA | CT SP 21 | 2008/06/16 John Practice [Signed CN number ooxxxforRevisionA Specifications. 
[19] ProgA | CMT SP24 | 2005/09/01 John Practice _—«[CCBto analyzethe RevisionACN. 
20], ProgA [| CMT SP214 | 2005/09/01 John Practice [File Version Histoy View(PDM) 
21] ProgA | CMT SP27 | 2005/06/24 Joe Procedure [EMail SchedulingSoftware CCBAgenda 
j22|ProgA | CMT SP21 | 2005/06/24 Joe Procedure _—‘|SoftwareCCB Minutes:DatedTBD 
23] ProgA | CMT SP21 | 2005/06/24 Joe Procedure —|SoftwareCI'sfromSDP 
[24| ProgA | CMT SP214 | 2005/06/24 Joe Procedure _—*([Softwae CRs 


25] ProoA | CM | SP22 2005/09/01 John Practice Change incorporation history (screen shot of PDM fromRevision - to Revision A) 
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Tracking Matrix Output #3 Raytheon 
Plans Completed 


Practice 
SP 1.1 Configuration Management Plan 


Program Management Plan (PMP) Program Management Plan (PMP) Measurement and Analysis Plan (PMP) 
Measurement and Analysis Plan (PMP) 
Measurement and Analysis Plan (PMP) 


Measurement and Analysis Plan (PMP) 


Program Management Plan (PMP) 


SP 1.2 Configuration Management Plan 


SP 1.3 


Program Management Plan (PMP) Program Management Plan (PMP) Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) Program Management Plan (PMP) 


SP 1.4 
SP 1.5 
SP 1.6 
SP 1.7 
SP 2.1 
SP 2.2 
SP 2.3 
SP 2.4 
SP 2.5 
SP 2.6 
SP 2.7 
SP 3.1 
SP 3.2 
SP 3.3 
SP 3.4 
SP 3.5 
GP 2.1 
GP 3.1 
GP 2.2 
GP 2.3 
GP 24 
GP 2.5 
GP 2.6 
GP 2.7 
GP 28 


Program Management Plan (PMP) Program Management Plan (PMP) 


Program Management Plan (PMP) Program Management Plan (PMP) 


Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) Measurement and Analysis Plan (PMP) 
Measurement and Analysis Plan (PMP) 
Measurement and Analysis Plan (PMP) 


Measurement and Analysis Plan (PMP) 


Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) 


Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) 
Program Management Plan (PMP) 
Program Management Plan (PMP) 
RMS Practice O7-04-525 


RMS Practice O7-04-525 RMS Practice O7-04-525 


Program Management Plan (PMP) 


RMS Practice O7-04-525 
Measurement and Analysis Plan (PMP) 
Measurement and Analysis Plan (PMP) 


RMS Practice O7-04-525 


Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) Program Management Plan (PMP) 


Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) Measurement and Analysis Plan (PMP) 


Training Plan 


Program Management Plan (PMP) 


Training Plan Training Plan Training Plan Training Plan 


Configuration Management Plan Data Management Plan Dats Management Plan 
Stakcholder Involvement Plan (PMP) Stakeholder Involvement Plan (PMP) Stakeholder Involvement Plan (PMP) 


Measurement and Analysis Plan (PMP) Measurement and Analysis Plan (PMP) Measurement and Analysis Plan (PMP) Measurement and Analysis Plan (PMP) Measurement and Analysis Plan (PMP) 


GP 2.9 Interceptor Objective Evaluation Plan (OE) | Interceptor Objective Evaluation Plan (OE) | Interceptor Objective Evaluation Plan (OE) | Interceptor Objective Evaluation Plan (OE) | Interceptor Objective Evaluation Plan (OE) 
GP 2.10 Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) Measurement and Analysis Plan (PMP) Program Management Plan (PMP) 
GP 3.2 Configuration Management Plan Program Management Plan (PMP) Program Management Plan (PMP) Measurement and Analysis Plan (PMP) Program Management Plan (PMP) 


Data Management Plan Data Management Plan 


Stakcholder Involvement Plan (PMP) 


Program Management Plan (PMP) 


> >I}, Plans Status (by Program) / CMMI Evidence Overdue Report (P_ / Activities Comp Status (by Prog 


Ake) (ce mOlelvelels 


© 2000, Raytheon Company. All Rights Reserved. 
Revised: September, 2002 - 20446AGP 1-24 


Tracking Matrix Output #4 Raytheon 
Assessment Coverage 


Blue = Fully 

Green = Largely wio Partials 
Yellow = Largely with Partials 
Partially or Not Satisfied 


Appraisal team 
knew precisely 
where they where 
with respect to 
required evidence 
using the products 
from the project 


O2FOHOHOOOOOONHHHHHHHOHHODHOHHHHHHD 
VIVPVVVVVVVVUVVVUVVUVVVVVVVUVUUUUS 
BRN NNNNNNNWNWWW 


PMC VAL | VER Ts 
Practice 
P14. 
P12 
1.3 


O2QHHHHHHHOOOOHVHHHHHOHHHHHHHHHHOHH 
VDIUVVVVVVVUVUVDUVUVUVUVUVUVUUUUNY 
ONY NNNNNNNWN WWW 
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Achieve Raytheon 


¢ Documented project using Specialists Template & STS 

¢ Rewarded stakeholders with gift certificates 

¢ Showed results of tracking matrix to outside assessors who Say it is Best in Class 
for tracking progress 

¢ Intend to present this approach at R6 Sigma Forum as an example of a “Go Fast” 
project that had results 

° Deliverables 
— Updated Deployment Tracking Spreadsheet 
— Updated Run Rules & Process Flow 
— Reporting Charts 

Deployment & Evidence Rate Charts (1 per Program & Roll up) 

Program Plans / TDs by PA and Practice (1 per Program) 

Program Opportunities by Program & PA 

Evidence Status by Program & PA 

Program Evidence by PA & SP (1 per Program) 

Program Evidence by PA & GP (1 per Program) 

— Demonstrated accurate information usable by the CMMI Project Team 

— All deliverables were generated on time with minimal changes 
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Lessons Learned Raytheon 


¢ Imposing strict run rules and enforcing them worked in the solution 


¢ Inputs from stakeholders & using them in the solution created a win / win 
for most parties 


e If you measure it, you should act on it 


— Less panic in the end game because we knew exactly where we were 
with the evidence 


¢ All SCAMPI A goals appraised as satisfactory 


Maturity Level 
Level 2 Level 3 Level 4 Level5 


q|3| 8| § 


|= @) 
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Raytheon 
Goal Attained 


Raytheon cumi | 
1 

A CMMI® based Appraisal was completed on December 16, 2004 | 
in accordance with the Software Engineering Institute's | 
SCAMPP™, V1.1: Method Definition Document and it was 


determined that 


Raytheon Missile Systems 


SEI Level 3 Process Maturity 


as defined by the SEI CMMI® Version 1.1 
SE/SW Staged Representation. 


(signed) (signed) 
Albert J. Truesdale James Armstrong 
SEI Authorized Lead Appraiser Appraisal Team Member 
Center for Systems Management Software Productivity Consortium 
“= SOFTWARE 
Poebles ig OR itt => PRODUCTIVITY 
larg Yigae CONSORTILIM 


MANAGEMENT 
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The Continuous Representation: 


Cultivating Process Improvement from 
Grassroots Efforts 


November 14, 2006 


Jennifer Turgeon 
Sandia National Laboratories 


2. 


TAT aa Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, Sandia 
UM NG A" = for the United States Department of Energy’s National Nuclear Security Administration National 
———— under contract DE-AC04-94AL85000. Laboratories 


g 


ot 


¢e My Background 
¢Overcoming Objections to CMMI 
¢ Know the Problems 


¢ Staged vs. Continuous for Small Organizations 
and Project Teams 


¢ Small Project Teams 
¢ The Plan 
¢Questions? 


Agenda 


Sandia 
National 
2 Laboratories 


peer 


My CMMI Background 


¢ Member of SCAMPIB & C Development Team 


¢ Member of Honeywell Aerospace and Corporate 
IT Appraisal Teams 


¢e Authorized SCAMPI B & C Lead Appraiser 
¢ Candidate Authorized SCAMPI A Lead Appraiser 


("h) Sandia 
National 
3 Laboratories 


CMMI Objections 


e Used inappropriately, 
ANY process 
improvement initiative ice 
can cause these 
problems. 


¢ The key is to focus on | 
what benefits the team : 
or organization the : 
most. 


N/ 


e CMMI creates endless 
documentation and 
useless artifacts 


¢ CMMI is only for large 
organizations and projects 


e CMMI interferes with 
creativity 


¢ CMMI costs too much 
¢ CMMI is too large and 
complex 


Sandia 
National 
4 Laboratories 


Ag 


Overcoming CMMI Objections 


e Listen to and document objections 
— Understand what the apprehension is 
— Address the issues directly 


— Communicate with and involve management in 
these discussions 


¢e Consider process improvement over a “Level” 
— Discuss the pros and cons with management 
— Realize that the “Level” can always come later 


Sandia 
National 
5 Laboratories 


4 


— What I’m doing works fine, why change? 


— | don’t have time to learn something new, what’s in it 
for me? 


— | don’t trust management’s motivation for change! 


— If | change, that means what I’ve been 
doing in the past was wrong... 


—| don’t want to move out of my comfort | 
zone... 


— | have no control over what Is 


changing... 
(7h) ise 
6 Lahotatovies 


Know the Problems 


Staged vs. Continuous 


For small organizations and project teams... 


Staged 
«Can be costly, especially 
when Maturity Levels are 
selected arbitrarily 


¢ Overwhelming for people 


new to process 
improvement 

¢ Can lead to process 
improvement gone 


BAD 


Continuous 

¢ Cost effective; 
organizations can prioritize 
and where they need to 
improve systematically 

e Reduces anxiety by 
focusing on limited areas 

¢ Supports the need for 
institutionalization 


Sandia 
National 
Laboratories 


Presentation Scope 


Small organizations — Team size of 1-5 people 
and teams — New to defined or 
: consistent use of 
rere process 


— Have a general desire 
to improve the work 
that they do 


— Have some undefined 
processes in place that 
are unique to each 
team member 


— Averse to CMMI 


("h) Sandia 
National 
8 Laboratories 


peer 


The Plan 


Let the team build the processes 


("h) National _ 
Laboratories 


ap 


¢ Find process improvement grassroots advocates 


— These people believe that their work can be 
improved and are willing to do something to 
change 


— They generally do not like “initiatives” because 
they have seen these come and go 


— They are leaders in their work community 
¢eLet the advocates drive the change 
— Other practitioners trust these advocates 
— They understand how to express the need for 
change to their peers 
10 (th) ia 


Bottom Up/Grassroots Approach 


A 


Give the Implementation Team Control 


¢e Discuss and identify benefits that can be 
achieved from change 
— Identify benefits to the team members 
— Identify benefits to the customers 
— Identify benefits to the organization 
¢ Identify areas for improvement 
— Teams identify areas for improvement 
— Teams prioritize the identified areas 
— Teams then schedule the improvement efforts 


ationa 
11 Laboratories 


_— 


eg 


Small Steps 


¢ Develop a schedule that works for the 


implementers 
—Keep management informed 
— Allow for small increments of change 


¢ Small steps that elicit great success will allow 
for bigger steps the next time around 


— Hold people accountable for their work 
— Communicate frequently 


ationa 
12 Laboratories 


_— 


eA~4 


Start with Existing Processes 


¢ Identify existing processes 
— Provide time for teams to share what they do 


— Include both documented and undocumented 
processes 


— Take the time to understand the underlying 
process culture 


¢e Discuss how existing processes can be 
improved 
— Attempt to maintain the culture as processes are 


improved 
("h) Seles 
13 presber 


= 


por 


Simple Steps to Process Improvement 


¢ Define what you do (may be the draft of a process) 
¢Ildentify areas for improvement 

¢ Establish goals 

¢ Develop a process (or refine what you already have) 
¢ Use the process 


¢eReassess how you are doing after using the new 
process 


¢ Continuously improve, as needed 
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Using the Continuous Representation 


¢ Use the continuous representation as a process 
improvement guide 


— Provides a mechanism for measuring improvement 
toward goals 


— Allows management to identify readiness fora 
formalized appraisal 


¢ Avoid “reinventing the wheel” 


ationa 
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Continuous Representation 


* Requirements Management 
+ Requirements Development 
* Technical Solution 

+ Product Integration 

* Verification 

* Validation 


iona 
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Continuous Generic Goals 


Generic Goals _Generic Practices 
GG1: Achieve Specific | GP 1.1: Perform Base Practices 
Goals 


GG2: Institutionalizea | GP 2.1: Establish an Organizational Policy 
Managed Process GP 2.2: Plan the Process 
GP 2.3: Provide Resources 
GP 2.4: Assign Responsibility 
GP 2.5: Train People 
GP 2.6: Manage Configurations 
GP 2.7: Identify and Involve Relevant Stakeholders 
GP 2.8: Monitor and Control the Process 
GP 2.9: Objectively Evaluate Adherence 
GP 2.10: Review Status with Higher Level Management 


-GG3: Institutionalize a_| GP 3.1; Establish a Defined Process 
Defined Process GP 3.2: Collect Improvement Information 


GG4: Institutionalize a | GP 4.1: Establish Quantitative Objectives for the Process 
Quantitatively Managed | GP 4.2: Stabilize Subprocess Performance 
Process 


GG5: Institutionalize an | GP 5.1: Ensure Continuous Process Improvement (fh) Sandia 


Optimizing Process =| GP 5.2: Correct Root Causes of Problems phen 


Simple Steps to Process Improvement 


¢ Define what you do (may be the draft of a process) 

¢ Identify areas for improvement 

¢e Establish goals 

¢ Develop a process (or refine what you already have) 
¢ Use the process 


e Reassess how you are doing after using the new 
process 


¢e Continuously improve, as needed 
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Define What you Do 


Process Management 
* Organizational Process Focus 
* Organizational Process Definition 
‘(Organizational Training 
* Organizational Process Performance 
* Organizational Innovation and Deployment 


Product Management 


Engineerinc 
«Requirements Management 
+ Requirements Development 
* Technical Solution 


* Product Integration 


‘Werfication 
‘Nalidation 


Support 


Sandia 
National 
Laboratories 
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Simple Steps to Process Improvement 


¢ Define what you do (may be the draft of a process) 
¢ldentify areas for improvement 

¢e Establish goals 

¢e Develop a process (or refine what you already have) 
¢ Use the process 


eReassess how you are doing after using the new 
process 


¢e Continuously improve, as needed 
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Identify Areas for Improvement 


Process Management 
* Organizational Process Focus 
* Organizational Process Definition 
‘(Organizational Training 
* Organizational Process Performance 
* Organizational Innovation and Deployment 


Product Management 


Engineerinc 
«Requirements Management 
+ Requirements Development 
* Technical Solution 


* Product Integration 


e. 
) 


se 
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Support 


Sandia 
National 
Laboratories 
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Prioritize Improvement Areas 


_Process Management Engineering 
* Organizational Process Focus +/Requirements Management 


+ Organizational Process Definition + Requirements Development 
‘(Organizational Training = * Technical Solution 
* Organizational Process Performance * Product Integration 
* Organizational Innovation and Deployment 1" Verification 
* Validation 


Support 


Product Management 
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Simple Steps to Process Improvement 

¢ Define what you do (may be the draft of a process) 

¢ Identify areas for improvement 

¢ Establish goals 

¢ Develop a process (or refine what you already have) 


¢ Use the process 


eReassess how you are doing after using the new 
process 


¢e Continuously improve, as needed 
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Establish Goals - 1 


Process and Product Quality Assurance Process Area 


¢ SG 2 Provide Objective Insight 


— SP 2.1-1 Communicate and Ensure Resolution of 
Noncompliance Issues 


— SP 2.2-1 Establish Records 


ationa 
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¢ Goals may be all or some of the specific practices 
¢ Be careful with the jargon 
— “We are working toward improving our 
functional testing” versus 
— “We are implementing SPs 1.1 - 1.3 in order to 
improve the way that we prepare for functional 
testing 


eit is important to pace achievement towards goals 
to ensure success 


Establish Goals - 2 


ationa 
25 Laboratories 


Generic Goals 


GG1: Achieve Specific 
Goals 


GG2: Institutionalize a 


Managed Process 


GG3: Institutionalize a 
Defined Process 


GGA: Institutionalize a 
Quantitatively Managed 
| Process 


GG5: Institutionalize an | 


| Optimizing Process 


Establish Goals - 3 


Generic Practices 


GP 1.1: Perform Base Practices 


: Establish an Organizational Policy 
: Plan the Process 
: Provide Resources 
: Assign Responsibility 
: Train People 
: Manage Configurations 
: Identify and Involve Relevant Stakeholders 
: Monitor and Control the Process 

.9: Objectively Evaluate Adherence 

GP 2.10: Review Status with Higher Level Management 


GP 3.1: Establish a Defined Process 


GP 3.2: Collect Improvement Information 


GP 4,1: Establish Quantitative Objectives for the Process 
GP 4.2: Stabilize Subprocess Performance 


GP 5.1: Ensure Continuous Process Improvement 


GP 5.2: Correct Root Causes of Problems 


Sandia 
National 
Laboratories 


a 


Establish Goals - 4 


¢Generic practices start to take care of 
themselves as processes improve 
—Monitoring generic practices should bea 
project management activity reported to 
management 
«Don’t talk CMMI until practitioners are 
ready 
— Gradually introduce the fact that the 
process improvement efforts align well with 
CMMI implementation 
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Simple Steps to Process Improvement 


¢ Define what you do (may be the draft of a process) 

¢ Identify areas for improvement 

¢ Establish goals 

¢ Develop a process (or refine what you already have) 
¢ Use the process 


eReassess how you are doing after using the new 
process 


¢e Continuously improve, as needed 
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Develop a Process 


¢eRelevant practitioners need to write, or at 
least edit, the process 


¢Keep the process simple, even if it is not 
complete 

¢Don’t shoot for a perfect process the first 
time 

«Get team consensus 
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Simple Steps to Process Improvement 

¢ Define what you do (may be the draft of a process) 

¢ Identify areas for improvement 

¢e Establish goals 

¢ Develop a process (or refine what you already have) 


¢ Use the process 


e Reassess how you are doing after using the new 
process 


¢e Continuously improve, as needed 
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¢Use the process for an extended period of 
time 

«Collect issues related to the process as 
they arise 

eReassess how you are doing after using 
the new process 


Use the Process 


¢Continuously improve, as needed 


ationa 
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Contact Information 


Jennifer Turgeon 


jlturge@sandia.gov 
505-284-3630 
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Questions? 


LOCKHEED wantin Ty 


CMMI®-Based Improvement: 
The View from the Project 


Beth Clark 


Trish Cavenee 
Lockheed Martin Integrated Systems and Solutions 


The Project ae 


¢ Large-scale Ground System that fits into larger, 
-S) of-Tor-rs ofekey-re me) cole} e-lanle e) mend ellale m=t-) abso) me l-lc. lm Ce) 
military users 


— Software-intensive, net-centric environment 
— Part of larger, system of systems 


DM al-mO) ae rlalyczchiveya) He 


¢ Lockheed Martin Integrated Systems & Solutions, 
which achieved CMMI® Version 1.1, 
SE/SWI/IPPD/SS Maturity Level 5, October 2005 
¢ IS&S has rigorous CMMI® maintenance 
methodology 
— All programs visited 
¢ 6-9 months after ATP with SCAMPI™ C 
¢ One year after initial benchmark, SCAMPI™ B 
'o) a Oar ol=) ol-JalollaleMelamccUelcmclalel(e) mu atsy.4 
om X=) ol-1alellare mela e)aeler-t-t-m al). @omle(-laldiil-ce Mla] 
previous benchmarks, revisited at least every 
two years 


The Project Process History a 


¢ Awarded in late 1990’s 
J col) o) (Toi ap merey=) ar-lalele-yes a [-LolUI(-m ogo) eo) (-1p nhs) 


¢ In 2001, initial SCAMPIS” C held to assess process 
risk — declared “red” 


¢ Process Action Team formed and improvement 
strategy defined 


— Process initiatives managed like a project 


— Benchmark at least annually, until “blue” risk 
chal ale ireYou a) (-\V(=Yo| 


Project Improvement Strategy 


¢ Generate How’s and What’s, Then Assess 
¢ Repeat until Blue 
¢ Maintain 


What Did We Do? He 


¢ Aligned project processes with organization’s 
standard processes 

¢ Provided adequate resources (training, tools, 
FeVey=)] 

¢ Instituted quantitative management and causal 
analysis 

a =T-1aleralait-la.<-rem ¢) more) e-lanm elgeler-c-som irc helaiavarclalalercli hye 
olor alaliale Mlam40eyzmm ce) Ce) Tale i box<oom Oa) 11 | be 
maintenance methodology 

¢ In 2006, participated in IS&S SCAMPIS A 


Process Improvement Results ae 


¢ Better and better benchmarks 
¢ Measurable improvement in project execution 
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‘Actual Productivity 
Productivity Goals 
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© SCAMPI SCAMPI SCAMPI SCAMPI 
2002 2003 2004 2005 2006 
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Reqs & Design 


Internal Test 


Lifecycle Phase 


{ 


PRODUCT A 


System Test Customer 
Deployment 


What We’re Doing Now =e 


Oro) alilaleiiale Mcomelarel-e-jic-lavem-lalem lanl e)geh—m ol geler-t-t= 
performance capabilities 
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deployment 


Questions? 


—S——. Carnegie Mellon 
“== Software Engineering Institute 


Pittsburgh, PA 15213-3890 


Panel: The Business Value 
Of CMMI ML 5 


Lynn Penn Lockheed Martin 
Rushby Craig Hill AFB 

Rick Heffner Northrop Grumman 
Gene Miluk SEI 


Sponsored by the U.S. Department of Defense 
© 2006 by Carnegie Mellon University 


Version # Course or Lecture or Module Info. - page 1 


—>—.. Carnegie Mellon 
Software Engineering Institute 


CMMI Maturity Level 5 


eDescribe CMMI ML 5 beyond OPP, QPM, OID and CAR 
eHow do you operationalize ML5 

eWhat do you measure? What do you watch? 

eWhat business value results: 


- For your customers 
- For your organization 


LOCKHEED MARTIN A 


Lockheed Martin 
Integrated Systems & Solutions 


CMMI Maturity Level 5 Foundation 


M. Lynn Penn 
Director Quality Systems & Process Management 


Integrated Systems & Solutions 
Process Improvement Structure ‘°°«*"©5©° ” carne 


Integrated Systems & Solutions 
S. Sloane, Ex Vice President IS&S EPSC 


VP Program & Executive Process PPS 

Mission Success Steering Committee PP Template 

P. Blow-Mitchell PPS Compliance Matrix 

Required Process Orientation Training 


Lynn Penn,-Chair 


/ Engineering : 


| Horas Deenend Program Management 
i i fof) teed Business Management 
Engineerin song 
CORPORATE EPI ee | Sg, Mensanasense Business Operations 
TECH OPS Vice President g Masetainniannen Operations 
MGMT : 
COUNCIL 


Representatives then ask | Approval of Program Specific 
RESP RING Z0ne IO REIN PPS Compliance and Program 
CORPORATE EPI Representative ; Plans 
SYSTEMS 


SUBCOUNCIL 


; Engineering Procedures & Procedure Change Approval 


Alternates . : 
Engineering Process Procedure Enhancements 


Improvement Metrics, Reuse, ... 
Training Courses & Materials 


CORPORATE EPI Engineering Process Asset Library 


TEAM 


Process Roadmap 


IS&S IS&S 
Process Quality 
Directive Manual 


LOCKHEED MARTIN A. 


Process Documentation Process 
Improvement 


Audit 
Programs 


Continuous 
Process 


Manuals 
Improvement 


Tailoring Program 
Guidebooks Profile 


Database 


IS&S 
Process 
Asset 
Library 


OOmi-053 


Communication... Training...Communication...Training 


LOCKHEED MARTIN 4 


IS&S Measurement Program 


Objective: The IS&S Measurement Program provides the 
infrastructure, data, and analysis that 
¢ facilitates program level quantitative management, 
¢ establishes the organization’s process performance baselines, 
¢ facilitates the alignment of organizational process performance 
with business objectives, 
¢ identifies opportunities to improve program and/or organizational 
processes, 
° oe program performance to establish deployment initiatives, 
an 
¢ facilitates cost, schedule, and quality estimation. 
Led by: |S&S Measurement Coordinator 
Reports to: Director, Quality Systems & Process Management, 
Mission Success 
Oversight: Measurement Program Steering Committee 


LOCKHEED MARTIN A. 


IS&S Approach to CMMI Measurement 
Requirements 


Focus CMMI 


Command Media Planning 
Process Area 


Program Quantitative IPS0023 
Project Quantitative 
Management Management 


PPS 2.8 
Quantitative 
Management 


Determine 
Information 
Needs 


Select 
Indicators 


Determine 
Decision 
Criteria 


Quantitative Fundamental Care Engineering 
Management Information indicator 
Manual Needs 


Measurement 
Report 


Organizational Organizational IBP0023 Collect Analysis 
Process Measurement 
Data Sets 
Performance Program 


Determine Core Indicator 
Performance Objectives 


LOCKHEED MARTIN A. 


Quantitative Project Management 


The process to the right is 
quantiatively managed using 
the indicator below 
Earned Value, Schedule 
Earned Value, Cost 
Product Progress 
Requirements Traceability 
Requirements Stability 
TBDs and TBRs 

Design Stability 

Breadth of Testing 

Monthly Fault Profiles 
Cumulative Fault Profiles 
Defect Density 

Staffing 

Product Growth and Progress 
In-process Productivity 
Defect Detection Profile 
Effort Profile 

Technical Performance 
Measurements 


The critical subprocess to the 
right is statistically managed 
using the indicators below 
Preparation Rate Control Chart 
Pace Rate Control Chart 
Defect Density Control Chart 


LOCKHEED MARTIN A. 


Organizational Measurement Analysis 


Type Residence Access 


Raw Data IS&S Measurement Measurement 
(Data Sets from Programs) eepostipry UME), Pro or am 
Program Summary Proposals 
Cross Program Analysi All IS&S 


EMR: Engineering Measurement Report PPD: Program Profile Database 


IS&S | ourney to CMMI L5 
Program Performance 


Software Productivity Up 52% e 95.5% of programs >) 
Potential Additional ener $50M have no serious 
Award Fee Achieved : problems. 


Down 23% 
Down 23% ¢ Improved Award Fee 


performance 
forey aia l alesse 


Process Maturity 


S/W productivity increased 52% S/W cost decreased 23% in Defect Find / Fix cost 
over baseline average capability Constant 2004 dollars down by 21% 


|Molow se aleltle 


Relative $ / Defect 


a || 


istori CMMI L5 istori 
alissteyatersl| alisieyatercl| CMMI L5 rplsyteyatersl| CMMI LS 


Drive World Class Performance / Competitiveness 


LOCKHEED MARTIN A, 


IS&S Benchmarking Maintenance 


Initial Benchmark 
¢ SCAMPI C 6-9 months after ATP 


One year after Initial Benchmark: 


SCAMPI B 
¢ All programs over $X contract value 
¢ Some programs between $X and $X 


— FFP, numerous subs, not co-located team, high risk, 
corporate or IS&S visibility 


SCAMPI C 
¢ Non-SCAMPI B programs between $X and $X 
¢ All programs (>$X) that are following the “primes” or 
“customers” processes 


— A PPS mapping is required and this is done for process 
risk determination 


REVISITS depend on “Process Risks” identified in previous 
benchmark — ALL PROGRAMS VISITED EVERY TWO YEARS 


LOCKHEED gierie 
The Challenge 


Oo S ce? 
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PROCESS COMPLIANCE 
PROCESS PERFORMANCE 


LOCKHEED MARTIN A 


Questions ? 


NORTHROP GRUMMAN 


DEFINING THE FUTURE 


— Busi ness Value of 
me! R=\V/~) Be 


CMMI HM -Yeq a} ate) (ole hVm Oxo) a(-1a-1a[ei-mmOl-1-1 mC] ge] 0] 0) 
13-16 November 2006 


a Rick Hefner, Ph.D. 
Director, Process Management 
De Colatalae) Mel avlialiar-lamere)gelele-iceyal 
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Two Complimentary Approaches 
to Process | mprovement 


= 4 


Ss 


, 


4] 


= Clarify what your customer 
wants (Voice of Customer) 


= Critical to Quality (CTQs) 
= Determine what your processes 
can do (Voice of Process) 
= Statistical Process Control 
= Identify and prioritize 
improvement opportunities 
= Causal analysis of data 


= Determine where your 
customers/competitors are 
going (Voice of Business) 


= Design for Six Sigma 


Model-Driven (e.g., CMM, CMMI) 


Determine the industry best 
practice 


=" Benchmarking, models 


Compare your current practices 
to the model 


= Appraisal, education 
Identify and prioritize __ 
improvement opportunities 

= Implementation 

= Institutionalization 


Look for ways to optimize the 
processes 


NORTHROP GRUMMAN 


Lean Six Sigma Provides the Needed Tools 
to Implement CMMI High Maturity 


7) 
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Number of No: 


Process variation Process performance 


UCL 
10 


8 
6 
4 
2 
0 

Projects Audited in First Quarter 


Stabilize 


Level 4 


Understand project’s process 
capabilities 

based on process performance 
baselines 


Control process variation 
(remove “assignable causes”) 


Predict results using process 
performance models 


Manage to achieve goals 


Process knowledge 


Predict Tan) ol ge)(cmr.am Oxo) alice) 


Level 5 


= Base improvement goals on 
future business needs 


= Eliminate problem and defect 
causes (“common causes”) 


= Select, predict, and measure 
improvements to change the 
process performance baselines - 
shift the mean; tighten the variance 


= Manage change 


NORTHROP GRUMMAN 
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Barriers and Challenges 


= Engineering process measurements are often difficult to analyze 
— / = Inherent process variations when human creativity is involved 
: = Dirty (or no) data 
= Vague measurement definitions, human recording errors 
= Infrequent measurements 
= Non-normal data 
= Need for stratification/aggregation 


= Must demonstrate the value of quantitative data to managers 
= Management style - reactive vs. proactive vs. quantitative 
= Less value in a chaotic environment 
= Must involve customers 


NORTHROP GRUMMAN 
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¢ Finalize project 


Launch 0 


goals 

Executive ree 

Overview measurement 
data for the 


subprocesses the 
project will put 
under SPC 
analysis 


Become familiar 
with Northrop 
Grumman Mission 
Systems QPM 
process 


Definition of 
related six sigma 
project(s) 


Incorporate and 
measure progress 
against the Level 
4/5 plan tasks 


Launch Workshop Strategy 


Collect relevant 
project historical 


data, review it and 


perform initial 
“clean up” 


Collect data from 
project 
subprocesses 
and put on SPC 
charts 


Complete project 
profile 


Review business 
issues for clarity 


Identify 
Quantitative 
Measurement 
Plan data 


Measure progress 


against the Level 
4/5 plan tasks 


¢ Finish building 
models 


¢ Begin tracking to 
models 


¢ Draft Quantitative 
Measurement 
Plan 


¢ Draft QPM 
evidence 


¢ Measure progress 
against the Level 
4/5 plan tasks 


¢ Develop project 
CAR plan 


¢ Start project CAR 
activities 


¢ Submit 
improvements to 
support OID 
(when applicable) 


¢ Measure progress 
against the Level 
4/5 plan tasks 


NORTHROP GRUMMAN 
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How Does Level 4 & 5 Benefit the Customer? 


Le 


Organizational 
process performance 


= Problem behaviors are 
recognized faster, enabling 
quicker resolution 


Quantitative project 
management 


Organizational 
Innovation and 
deployment 


= The project fixes the sou e 
Causal analysis of defects to prevent future / 
defects J 


dd 


Level 5 reduces costs and improves quality (so we implement it on all projects) 


Reference: How Does High Maturity Benefit the Customer?, NORTHROP GRUMMAN 


6 R. Hefner, Systems & Software Technology Conference, 2005 
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Lessons Learned 


*4 Based on over 20 Northrop Grumman CMMI Level 5 organizations 


=" Having multiple improvement initiatives helps encourage a 
change in behavior as opposed to “achieving a level” 


% = Reinforces that change (improvement) is a way of life 


= CMMI and Six Sigma compliment each other 
= CMMI can yield behaviors without benefit 


= Six Sigma improvements based solely on data may miss innovative 
improvements (assumes a local optimum) 


= The real ROI comes in institutionalizing local improvements 
across the wider organization 


= CMMI establishes the needed mechanisms 


= Training the staff as Six Sigma Green Belts has resulted ina 
change of language and culture 


= Voice of Customer, data-driven decisions, causal analysis, etc. 
= Better to use the tools in everyday work than to adopt the “religion” 
NORTHROP GRUMMAN 
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Ogden Air Logistics Center 


309 SMXG 


Business Value of 
High Maturity 


CMMI Technology 
Conference 


U.S. AIR FORCE a 


November 2006 


BE AMERICA’S BEST 


BS 54 Who We Are 
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OGDEN AIR LOGISTICS CENTER 


= 309‘ Software Maintenance Group, Hill AFB, Utah 
= Approximately 700 engineering personnel 
= Develop and maintain software for Air Force and 
other DoD customers 
= F-16 Block 30 OFPs 
= Mission Planning software 
a Aircraft and Missile Automatic Test Systems 
= Command and Control Software 
= IT software 
= Majority of work is software maintenance 
a3 product lines--containing 40+ projects total 


BE AMERICA’S BEST 


Y 


ALC | 


Provides focus 
for what we 
want to 


Our High Maturity Strategy oo 
309 + 


OGDEN AIR LOGISTICS CENTER 
Customers 
oer” ‘Needs: 
Wh Strategic Plan 
Metrics Implementation 


Guide 
Product 


«Standard 
Processes 
Quantitative 
Goals 
*Models 
-Baselines 


~—~< 


Operational Flight 


Weapon eras Software 
Software 


Automatic Test Systems 


BE AMERICA’S BEST 


: § Q 
Product Lines gy 


OGDEN AIR LOGISTICS CENTER 


aln addition to an organization-level SEPG we have 
Extended SEPGs (ESEPGs) in each product line 


= ESEPG leads are: 
= Accomplished Project Managers 
= Responsible for mentoring new and less experienced 
Ms 


= Responsible for leading process improvement in the 
product-line 


¢eEstablishment of process baselines, and models and 
ensuring their use 


«Collection of measures 
eLeading CAR teams 


= ESEPG members typically made up of PMs and 
project leads from within the product line 


BE AMERICA’S BEST 


Benefits 1 
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= Improved Customer Relationships 


= Some customers were not initially supportive of our 
process improvement efforts 


= Now many customers can quantify their expectations 
to us 


= They are comfortable with reviewing project data and 
even understand it 


=a They have more realistic expectations 


= They can see that they are getting more product for 
their dollar than in the past 


BE AMERICA’S BEST 


Benefits 2 
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OGDEN AIR LOGISTICS CENTER 


Three fold reduction in defect density while 
increasing Productivity by 60% (OFP) 


417% increase in productivity (GTACS) 


Cycle time reduced from 120 days to less than 60 
days in past 2 years. (ATE) 


Ln 
BE AMERICA’S BEST 
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Mike Ballard mike.ballard@hill.af.mil 
Rushby Craig rushby.craig@hill.af.mil 


SEPG 
309 Software Maintenance Group 
Hill AFB, Utah 


BE AMERICA’S BEST 
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DEFINING THE FUTURE 


~SCAMPI A Appraisals = 
Preparation In 100 
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_CMMI Technology Conference & User Group 
_ 13-17 November 2006 


% Ron Ulrich 
ae Manager, Appraisal Program 
_ Mission Systems Sector 
4 MNorthrop Grumman Corporation 
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What’s ahead 


= You can save $$ and still be effective 


«= *) * Prepare for the appraisal 
. = Artifacts 
= Optimize to save 


= Training 
= Consistency 
= Presentations 
= Lead Appraiser Selection 
= Appraisal Plan/Input 


= Schedule 
= Optimize to save 


= Measure & analyze 
= Deliver the results to the sponsor 


NORTHROP GRUMMAN 


CMMI Background Message - 1/ 2 


= Northrop Grumman Corporation 
= 7 Sectors (including Mission Systems) 


» 4 = 79 SCAMPI A’s - 52 externally led 
_. { = 28 Level 5 SCAMPI A Appraisals 
» = 15 Authorized SCAMPI A Lead Appraisers 
' + * Northrop Grumman Mission Systems 


. = Started June 2002 - first SCAMPI A Appraisal 
= Common, tailored Policies & Plans for all of Mission Systems 
= Performed over 57 SCAMPI A Appraisals (37 externally led) 
= 20 Level 5 SCAMPI A Appraisals 
= Appraised over 164 Projects (92 unique projects — annual re-appraisal) 
= 12 Authorized SCAMPI A Lead Appraisers 
= 4 Authorized “Introduction to CMMI” instructors 
= Measurements were taken on all appraisals & analysis done using Six Sigma 
= Time to review artifacts (average 2.0 hours per PA per Project) 
= Time to perform interviews (average 1.5-3.0 minutes per SP/GP with 3 projects or less) 


_ = Nice to work in an organization that supports CMMI, 6 Sigma and process 
J improvement (as most of you can imagine) 
= | have been on over 40 SCAMPI A Appraisals 


NORTHROP GRUMMAN 


CMMI Background Message - 


Northrop Grumman 
CMMI Level 5 Appraisals Performed 


(NOTE: Includes internally* led and "organizational repeats") 
(* Internally led appraisals are led by SEI Authorized SCAMPI Lead Appraisers who work for NG) 


‘Number [Organization _—_—~‘(|:—Levelibiscipiine [bate 
[28 iis-acsaoms CLS SWISENPPDISS | 2ANovOS 
[a7 ws-oms—SSCS~=~—~—SC~—~SSCi SES 
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[25 us-seospo———SOS~SSC SEIS —«dY—SC Ov 
[24 ms-mopicrco——SOS~S~S SSCs r+ —t-oet 0 
[23 ssc SSSCSC~—~SCS~S~SSC S| 
[22 [s-seomsO—SSSCS~S SCR Se SEY 
[21 |Ms-iPic (internaiiyiea) ———SC~SCSCi Sw *dY ayo 
[20 is-acsaems ———S Sn ss | 08-08 apron _| 
[19 |IT-DES (now Defense Group) =i is sis | 14-2amaros_| 
[18 [ws-nsopoas SS SSSSC*d;C*SSWLO'SE | 20-FeD-OS 
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€Time line for Level 5 SCAMPI A’s 
1, 9, 7, 11, 6 (2002-2006) 


e Each Organization MUST 
perform a SCAMPI appraisal 
ANNUALLY 

e Every other one MUST 
have an External Lead 
¢ All are tracked 
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Are you ready to be appraised? 


= What do you need? 
—/ = Process Improvement? 
3 = Level 2 — or higher? 


ss = If Level 3/4/5 does your organization have policies/processes in 
place? 

= You need to create them (or get them somehow) 

= Takes time if you need to do it from scratch 


= Do you know the details of all your projects? 


= You should have a memo or directive that specifies what projects 
apply to you business objectives (e.g., all projects over $500K, all 
projects with 10 or more personnel involved in CMMI processes) 

= You will need this to help your SCAMPI Appraisal Leader in 
selecting the appropriate projects for your organizational appraisal 


NORTHROP GRUMMAN 


Move out to get it done right 


= Details for the Scope of the Appraisal 
— = How many Projects? 
=" How many Appraisal Team Members? 
a = Is everyone trained? 
| = = You should have at least 1 member (but no more than half) from your 
: site 


= Need to do a “readiness review” 
» Why? 
= SCAMPI requires it ... and it makes you more likely to “pass” your 
goals 
= Alternate approach (details later) 


Perform the Appraisal 

= Select Lead Appraiser 
= Plan 

= Schedule 


= Follow on activities 
= Correct deficiencies 
= Do it again (keep projects involved) 
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Appraisal Process Flow 
(Pre Onsite SEPG Lead/Site Coordinator& Lead Appraiser**) 


Decision to assess organization 
Decide on Assessment Date & Select LA 
Select Project(s) & fill out PITD(s) 


$1 Begin Evidence Collection 

$2 Get Org Evidence Artifacts 

$3 Evaluate Evidence for assessment 
readiness 


Select Appraisal Interviewees 
Prepare Interviewees (brief & mock interviews) 


$4 Generate Appraisal Plan and Appraisal 
Input & Schedule 


Meet with Sponsor to review Appraisal Plan and Input 
Get Sponsor’s signature on Appraisal Plan and Input 
Develop Onsite Organization and Project Briefing 
Select Site Coordinator 

Select On Site Rooms (Main & 3 Interview Rooms) 
Setup for Onsite Internet Connections 

Provide Interviewee list to Lead Appraiser 

Get charge numbers for ATMs 

Get PIID Interview Questions with PIID responses 
Generate ATM Interview Notebooks 

Generate ATM Interview Notebooks 

Reproduce copies of Evidence Request Forms 

Set up interview rooms with mini-team notebooks 
Prepare Schedule and Submit to Site Coordinator (SC) 
Select Appraisal Team Members (ATMs) 

Prepare Team Briefing & Training 

Set up SEI TPWEB site for Assessment 


$ Talking points (later) 


** Order of Pre Onsite activities may performed in parallel 


$5 Establish Mini-team Assignments 

Send PAIS & other materials to SC for completion 

Consider NetMeeting for Findings Presentations 

Send out details of appraisal to ATMs 

Get any missing experience data from ATMs 

Modify any information in Schedule Templates and send to ATMs 
Perform Onsite ATM Training 


$6 Distribute Metrics Forms (collection times in 
worksheet) 


$7 Perform Interviews 
Lead consolidation activities 
Prepare Draft Findings 

Perform OU consolidation 


$8 Deliver Draft Findings Presentation 
Review Evidence as a result of Draft 

Perform Ratings to Goal, PA, Maturity Level 
Prepare Final Findings 

Deliver Executive Briefing 

Deliver Final Findings Briefing 

ATM Lessons Learned 

Gather metrics from team members 

Clean Up and Leave site 

Collect and destroy or maintain material 

Collect and maintain Action Items 

Prepare Process Improvement Plan 

Act on Process Improvement Plan 

Coordinate Actions of Process Improvement Plan 


$9 LA Analyze Measurements & Send Appraisal 
results to SEI Steward 
Get ready for Next Appraisal 
cael 


Two Approaches for Artifact Collection 
(get ready & cut costs - $1, $2, $3) 


Standard approach NGMS Approach 

1. Have projects and org gather 1.Specify Project “Book Bosses” 
artifacts for each Process Area (PA) 

2. Place into folders with (same person could do more 
annotation describing why the than 1) 
artifact was selected (NOTE: 2.Hire a consultant (Authorized 
annotation usually provides Lead Appraiser or qualified as 
little insight) such) 

3. Have qualified team review 3.Spend an hour training “book 
artifacts and specify which bosses” on what to gather for 
ones were wrong the Generic Practices 

4. Have projects and org gather 4.Have consultant and each PA 
replacement artifacts “book boss” meet for 2 hours to 

About 400 hours per project “gather, annotate, and capture” 


each artifact into folders 
About 100 hours per project 
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Some more Savings 1/2 


= Lay out your schedule* based upon metrics (take them too - $4, $6) 


F 4 = Don't forget 
all = Team meetings 
ay = Training** 
i _ a Appraisal team briefing 


= Site/org/project briefings 
= Plan for breaks (and don’t work late) 


= Assign mini-teams to review artifacts by PA category 
(e.g., engineering, Support, process management, project management, L4/5 — $5) 


= Almost everyone follows this approach (e.g. PPQA+CM, RM+RD) 
= Only trained qualified personnel on L4/5 team 
= Plan for 2 hours per process area per project (on average) 


" Mini-team alternative approaches 
= Review Projects PA’s as team (e.g. step thru each SP/GP) 
= GP’s assigned across all Projects (e.g. GP 2.1+2.9+3.1) 


- See Appendix A (typical schedule chart) 


e® Sec Hefner/Draper presentation - Applying CMMI Generic Practices with Good 
Judgment, http://www.dtic.mil/ndia/2004cmmi/CMMIT2Tue/1116RickHefner.pdf (2 NORTHROP GRUMMAN 
sample charts shown at end of this material) 


Some more Savings 2/2 


= Form interview sessions in parallel (i.e. mini-teams perform interviews 


s concurrently - $5,$7) 
—— = Questions for each mini-team to use 
* = Annotate/update questions based upon review of the artifacts 
; = Most interview sessions can be performed in about 90 minutes 


BUT more projects (e.g. over 3) take more time 
= Save about 1 day (analyze the measurements — $9) 


= Optimize the interview sessions (based upon number of projects) 
=" Specific Practices for ALL PA’s then GPs - $7 (30-45 mins/combo) 


= Draft Finding Presentation 


= During the team consolidation process you may find (if the org 
and project(s) are performing at the Level being appraised) that: 
= All artifacts have been returned (at least once) 
a = No significant weaknesses have been found 
= That the OU consolidation are all Fully or Largely 
Implemented 


= Save another day - $8 
NORTHROP GRUMMAN 


The Results 


= You can save $$ and still be effective 


= Prepare for the appraisal 
fr = Artifacts 
? = Optimize to save 
= Training 
= Consistency 
= Presentations 
= Lead Appraiser Selection 
= Appraisal Plan/Input 


= Schedule 
= Optimize to save 


_-—* Measure & analyze (se, 9) 
| = Deliver the results to the sponsor 
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APPENDIX A 


ey, Typical Appraisal 
Raw Data Collected 
During the 
Northrop Grumman 
Appraisal Process 


te 


__ Using the following data as collected from the pilots, estimates for interview times and for 
_ evidence evaluation times were calculated. Other factors (e.g. new person, Process Area, 
experience in PA, method of presentation) seem to influence the times. All are being 
investigated to further refine the NG SCAMPI Appraisal Process. 
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Data shown above is typical of collected data to support interview time spent during appraisals. 
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1 3 Average Time 95.975 


| Monday | 
Team Assembles 
Rm #### 
On-Site 
Training 
RM ##8# 


0830 
0845 
0900 


Opening Briefing 
BHEPNEWECSESLORSOE 


Main CR 
310-764-6338 Code: 


elt 


Working Lunch 


only 
RM ##8# 


ProjMgt Mini- 
Team 
PP, PMC 
RM ##8# 
YOOC 00-3000 
Code: AHH 


CT 
| 0800 _| 
| 0815 | 
0845 | 
| 0900 | 
| 0915 | 
| 9930_| 
0945 | 
_ 1000 | 
| 1030 _| 
| 1045 | 
1100 | 
| 1115 | 
| 1130_| 
| 1145 | 
_ 1200 | 
1215 | 
1230 _| 
| 1245 | 
_1300_| 
| 1315 | 
1330 | 


Whole Team 
OPF 
Rm #### 
aun-uat-nins Code: HH] Appraisal team 
Support Mini- 
Team 
OPD, OF 
RM ##8# 
WOE 00-3000 
Code: sR 
Consolidation 
Appraisal tearm 
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A Typical Schedule 


(SCAMPI A Level 5 with 3 Projects or less) 


Tuesday 
11/14/2006 


Team Meeting 
RM ##8# 


Working Lunch 


Appraisal team only 


RM #### 


Engr Mini-Team 
Pi, Val, 

Ver (testing) 
RM ##8# 
00300-30004 
Code: ARE 


Consolidation 
Appraisal tear 
only 


L4/5 Mini-Team 
QAM, CAR 


RM ##8# 
00300-30004 
Code: AAR 


Consolidation 
Appraisal ieam 
oniy 


L4/5 Mini-Team 
OPP, O/D 


RM ##8# 
OO }00-30004 
Code: AA 


Consolidation 
Appraisal team 
only 


ProjMgt Mini- 
Team 
SAM, IPM, Risk 
RM #### 
300-300-0006 
Code: AAR 


Consolidation 
Appraisal team 
only 


Consolidation 
Appraisal tearn 
only 


Consolidation 
Appraisal tear 
only 


Wednesday 


Engr Mini-Team 
RD, RM 


RM ##8# 
Y00C 3001-3000 
Code: AAR 


Consolidation 
Appraisal team 
onily 


Working Lunch 
Appraisal team only 
RM ##8# 


Engr Mini-Team 
TS, Ver (Peer 
Reviews) 
RM ##8# 
YOOC 300-3000 
Code: HAR 


Consolidation 
Appraisal tearn 
only 


Consolidate Findings 


Support Mini- 
Team 
M&A, DAR 
RM #### 
300-300-2000 
Code: AAR 


Consolidation 
Appraisal team 
only 


Support Mini- 
Team 
PPQA, CM 
RM #### 
300-300-2000 
Code: HH 


Consolidation 
Appraisal tearm 
only 


Draft Prelimin Findings in Mini-Tearms 


Appraisal tearn only 


Thursday 


| Friday 
Team Meeting 
RM ##8# 


= 
3 
rr 


Team Meeting 
RM #### 
Review 


Team Meeting 
RM ##8# 
Collect‘Review 


Draft Final Findings 
in Mini-Teams 


Working Lunch 
only 


Appraisal Team 
Wrap-Up 
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Samples from Hefner/Draper Charts 1: 


GP 2.6 - Manage Configurations Fa ET ET FR 


Place designated work products of the process under appropriate 
levels of configuration management. 


implementation - Must identify which work products are to be 
Guidance controlled and how they will be controlled (level of 
formality) 


— It is NOT necessary to designate and control every work 
product; omission implies “not controlled” 


Appraisal -A list of what is being controlled (e.g., a DM master 
Guidance document list) is not sufficient 
— Must understand what is planned to be controlled 


Aoplying CMMI Genenc Practices with Good Judgment Hefnes/Draper - 11 
2003 CMM! Technology Conference 17-20 November 2003 
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Samples from Hefner/Draper Charts 2/2 


GP 2.8 — Monitor and Control! the Process 


4s Monitor and control the process against the plan for performing the 
¥g process and take appropriate corrective action. 


implementation «Activities must be reviewed against plan, budget, and 
— Typically performed in staff meetings, cost/schedule 
reviews, variance reports 


~— Detailed review of the process performed against the 
process description is performed in the QA audits 


- Must establish appraiser consensus 
Guidance — Frequency, formality of review 
— Budgets need not be tracked to individual process areas, 
but each process area must be covered in some budget 
- Often difficult to find direct evidence that something 
is tracked 


— E.g., tracking of Project Management PAs 


Applying CMMI Generic Practices with Good Judgment Hefner/Draper - 13 
2003 CMM} Technology Conference 17-20 November 2003 
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Questions ? & Answers ! 
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4j Agenda Z : 


Introduction 

Approach 

Applying the Lockheed Martin 215' Century (LM21) 
Approach 

Improvements to SCAMPI Preparation and Conduct 


Summary and Next steps 


©Copyright 2006 Lockheed Martin Corporation 9 


11/14/2006 


SY Introduction Z : 


- Lockheed Martin typically conducts SCAMPI As at the enterprise level 
(3 Business Areas and 9 Business Units): 


= Corporate policy requires use of CMMI-SE/SW/IPPD/SS (will be CMMI- 
DEV+IPPD V1.2) 


= Promotes a consistent process context across the Business Unit 
= Ensures a unified approach to process improvement across the Business 
Unit 
= Facilitates integration for new acquisitions and business unit consolidations 
= Eliminates the overlap/waste resulting from conducting multiple SCAMPI 
A’s on the same set of organizational processes 
- Lockheed Martin Business Areas and Units are large organizations, 
managing geographically distributed, complex programs 
- Efficient SCAMPI preparation and conduct of SCAMPIs are required to 
ensure that they provide the best value for the business and the 
customer 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation 3 


K Yj Applying the LM21 Approach | re 


- Considerable resources are spent by LM Business 
Units to plan, prepare for, and conduct SCAMPIs 
- Lockheed Martin has employed its “Lockheed Martin 
215' Century (LM21)” approach to improve the 
process of appraisal preparation and conduct 
= Enterprise process improvement method 
- Benefits to customer and business 
= Reduced Business Unit cost to be appraised 
= Reduced impact to programs participating in appraisals 
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fax Lhe LM21 - Lean Processes that Operate at Six ae oe 
A Yj Capability 


Principles: 

~Value from the Customers 
perspective 

~Value Stream - measured 

—~Flow 

—Pull 

—Perfection 
Rapid feedback 
Mistake-proofing 


LM21 is the how-to guide for cutting 
cost and improving quality and 
customer satisfaction. 


It builds on the five principles of 

Lean as a set of leadership and 

decision-making mandates that 
define excellence: 


1. Customer Value — define it from the customer’s perspective. 
2. Value Stream — identify all activities used to make a product or provide a service. 


3. Flow — create a system where value is continuously added. Clear away obstacles 
that don’t add value or clog the value stream. 


4. Pull — initiate work only at the demand and to the specific specifications of the 
customer. 


5. Perfection — continuously refine the process to improve efficiency, cycle time, costs 
and quality. 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation 5 


REZ The LM21 Approach 


- This tool gives leadership a strategic look 
at its value streams and the ability to see 
waste at a macro level 


- Leaders identify and prioritize the 
improvement events necessary to 
= Eliminate the waste 
— Kaizen events (an activity where a team 
is chartered for a period of 3—5 days) 
= Identify waste for a given process and 
- Implement immediate, sustainable The result: a plan is in 
solutions for waste elimination/reduction place to strategically 


_~ Other VSMs, projects - activities that identify and eliminate the 
require extensive work and change waste that most interferes 


— Just Do Its - short term projects with the ability to deliver 
value to the customer. 


Consistently Deliven Value Growin Operating xcellencesss) 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation 


K Yi Applied VSM to the Appraisal Process Zy 


Formed a SCAMPI Tiger Team of process 
improvement leaders from across Lockheed Martin 
Business Units 


Performed a Value Stream Mapping 
Identified and evaluated candidate efficiencies 
Further defined the leading candidate efficiencies 
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©Y) Formed a SCAMPI Tiger Team A 


- Membership 
= Lockheed Martin Business Units with SCAMPI experience 


— Business Units had experience with a number of Lead 
Appraisers and Lead Appraiser organizations 


= Inputs represent a number of different approaches toward 
making SCAMPIs more efficient 
- Objective 
: Define the as-is processes, analyze them for inefficiencies, 
and identify opportunities for improvements that reduce 


— Business Unit CMMI appraisal preparation labor by at 
least 50% 


~ BU appraisal conduct costs by 25% 
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7 Performed a Value Stream Mapping (VSM) 4 


- Appraisal cost is dominated by preparation costs 


= Implementation artifacts from the organization and programs 
must be identified, verified, and characterized in the PIIDs 


= Not all artifacts are needed by the business 


- Efficient preparation is essential to maximize value 


ockheed Martin Policy 
(Enterprise Excellence) 


Process ._ 
< Value 23———> Improvement 


Maturity Level 


VSM is a Powerful Tool For Development of Strategic Performance Improvement Plans 
oameane ©Copyright ZUUo Lockheed Martin Corporation 


yj |mprovements to SCAMPI Preparation & Conduct Dy 


Legend: || - Prepare/Conduct Specific SCAMPI 


S| - Proposed Efficiencies 


SCAMPI A Preparation & Conduct 


Prepare Conduct Post- 
for Appraisal Appraisal 
Appraisal 
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NE Y Improvements to SCAMPI Preparation & Conduct Z 


Expected benefits: 


e Enable efficient, ongoing collection of data 

e Eliminate generating/collecting objective 
evidence of no value to the business 

e Support jump-starting new projects with 

templates for process artifacts 


Improvements: 
e Establish data structure & tools 


e¢ Plan optimal evidence collection and mapping 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation 1 


11/14/2006 


Sy Data Structure and Tools - 1 Z : 


Maintain a CMMI compliance data structure and 
Supporting tools to identify and manage all of the data 
necessary to prepare for and conduct a SCAMPI 
= Compliance data includes: CMMI, OSP, PDP, objective 
evidence, links 
Ideally, implement in a single database which is 
incrementally populated and verified 


= Distributes appraisal preparation effort as smoothly as 
possible across the interval between appraisals 


= Supports appraisal readiness by minimizing verification and 
update required in the immediate pre-appraisal period 


Implementation could vary: database report, web site, 
spreadsheets, etc. 


©Copyright 2006 Lockheed Martin Corporation 12 


Data Structure and Tool - 2 ez 


- Compliance data structure includes fields describing: 
= CMMI Model (PA, Goal, Practice) 
= Organizational Standard Process (OSP) and mappings to the CMMI 
= Direct and indirect evidence mapped to CMMI 


Compliance Data Mapping Concept 


CMMI Process Area OSP Project 
Goal 1 Defined Process 
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SY Data Structure and Tools - 3 Z : 


- Use the compliance data structure/tools to create pre- 
populated PIID templates for programs 


= Jump start the programs by providing them with PIID 
templates 


= Identify the objective evidence that would result if the 
Organizational Standard Process is performed exactly as 
documented 
- Make it easy for programs to document their specific 
work product information on an ongoing basis, e.g., 
= Program specific work product names 
= Location of the work products 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation 14 


7 Plan Optimal Evidence Collection & Mapping Zi 


= Eliminate generating or collecting objective evidence not 
required by the business 
= Eliminate redundant collection of objective evidence for 
Generic Practices 
— Generic Practices tend to be supported by Specific Practices, 
and in some cases, by other Generic Practices 
— Understand these relationships and identify where direct and 
indirect evidence that addresses the generic practices might 
already be produced 
= Efficiently use indirect artifacts 
— Determine what indirect artifacts are naturally produced by 
organization’s processes and map to the CMMI practices 
— Objective is to identify the minimal set of available indirect 
artifacts with a maximum correlation to corroborating the 
implementation of CMMI practices 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation ” 


SZ Improvements to SCAMPI Preparation & Conduct Z 


Expected benefits: 
¢ Collecting information incrementally throughout the 
project life cycle 
«Ils more efficient 
e Reduces program disruption 
e Supports sampling of programs across the 
enterprise 


Improvement: 

¢ Ongoing 
currency of 
mappings and 
evidence 
location 
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f Establish & Maintain neva 


Process & Objective Evidence Mappings 


- Maintain mapping of Organizational Standard Process 
(OSP) to CMMI practices (Specific and generic) 
= Use/populate compliance data structure and tools 


- Maintain mapping of Project Defined Process to 
CMMI practices (specific and generic) 
= Ensure that CMMI requirements are met at project startup 
and as changes are made to the OSP 
- Collect basic information on the project’s objective 
evidence on an ongoing basis: 
= As projects create their specific work products, they populate 
the compliance data structure and tools, e.g., 


— Program specific work product names 
— Location of the work products. 
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XN Yj Improvements to SCAMPI Preparation & Conduct ~ 


Expected benefits: 
¢ Provides efficient 
verification methods and 
tools 

¢ Improves consistent 
interpretation of requirements 


SCAMPI A Preparation & Conduct 


Efficient Objective 
Evidence Verification 


Improvements: 
¢ Verify evidence efficiently 


e Use incremental preparation approach 
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SY Efficient Objective Evidence Verification - 1 Z 


- Verify objective evidence efficiently 


= PIIDs must be carefully verified before an appraisal, including 
the links 


= Use asmall team (e.g. 3-5 people) 
= Use achecklist which documents standard criteria to improve 
consistency 
- Track progress and status of evidence collection 


= Collecting and verifying the objective evidence is time 
consuming and costly 

= Use compliance data structure and other tools to track the 
progress and status of evidence collection and evaluation 


11/14/2006 ©Copyright 2006 Lockheed Martin Corporation 19 


SY Efficient Objective Evidence Verification - 2 Z 


Perform independent verification of objective 
evidence with appropriate appraisal methods 


- Independently verify objective evidence before a 
SCAMPI A 


= Lockheed Martin recommends the Lockheed 
Martin Continuous Appraisal Method (LM CAM) 


~ CAM is an ARC Class B appraisal method 


~ CAM enables incremental appraisal and 
improvement 


= Maintain continuity of the appraisal team members 
between appraisal events 
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Summary of Improvements 


Legend: || - Prepare/Conduct Specific SCAMPI 


S| - Proposed Efficiencies 


SCAMPI A Preparation & Conduct 


Prepare Conduct Post- 
for Appraisal Appraisal 
Appraisal 
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\ 47 Summary and Next Steps | ZA 


- Summary 
= Efficient SCAMPI preparation and conduct is a business imperative 
= Improvements are expected to 
— Eliminate objective evidence not required by the business 
— Increase efficiency of data collection 
— Reduce program disruption 
— Support sampling of programs across the enterprise 


— Verify objective evidence efficiently (consistently, independently 
and incrementally) 


= Enterprise process improvement methods are being applied to the 
appraisal process improvement 
- Next Steps 
= Complete the estimated savings by implementation of improvements 


= Create a description of the “To Be” approach for SCAMPI 
preparation and conduct 


= Conduct pilots (underway) 
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Purpose: Describe Ongoing Research Meant To... 


1. Provide objective evidence to substantiate value added by CMMI- 
based process capabilities 


To what extent are there better performance outcomes when organizations 
& their projects follow processes that satisfy the goals of CMMI? 


2. Improve processes & resultant performance outcomes for 
organizations that participate in this research 


As well as for others that modify their own measurement activities similarly 
to support performance driven improvement 


3. Explain less than stellar product delivery by high maturity organizations 
as well as low 


As well as process enactment at the program/project level that does not 
always match appraised maturity 


Document the extent of such occurrences 
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Why is this Work Important? 


Substantial proof of concept exists 


- CMMI-based process improvement can & has led to concomitant 
improvement in performance outcomes 


- Predictably faster, better, cheaper product development and 
maintenance 


But skepticism remains about the value of disciplined adherence to 
well defined processes 


- As opposed to solutions de jour 


- That are not necessarily at odds with processes that satisfy the goals 
of CMMI best practices — e.g., Agile or Six Sigma methods 
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Some Existing Evidence 


Performance Results of CMMI-Based Process Improvement, Technical 
Report, CMU/SEI-2006-TR-004, August 2006 


Benchmarking CMMI Cost and Impact: Interim Report, December 
2004 (Distribution of full document limited to benchmark contributors) 


Demonstrating the Impact and Benefits of CMMI: An Update and 
Preliminary Results, Special Report, CMU/SEI-2003-SR-009, October 
2003 


CMMI Performance Results Web site 
http:/Awww.sei.cmu.edu/cmmi/results.html 
Numerous presentations 


At this & previous conferences 
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What’s Missing? 


Most existing work is based on case studies 
- Which are based on credible quantitative evidence 
- But circumstances differ 
— Case based results can be very instructive 
— But, they may not be applicable elsewhere 
— & they can be accused of “cherry picking” ... fairly or not 


More generalizable comparative analyses are needed ... Process models! 


- Of the effects on performance outcomes of differences in process 
enaciment 


- Under varying organizational circumstances & product characteristics 


— That may affect variation in both successful process enactment & 
performance outcomes 
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What Else is Needed? 


A viable benchmarking infrastructure & community of practice 
- Ina field where people aren't comfortable sharing information 


- Without which we're just guessing about “industry standards’ ... 


Empirical analyses that focus on barriers & facilitators of adoption & 
improvement initiatives 


- Process capability doesn’t always guarantee successful program 
performance & product quality 


- Not all improvement initiatives are implemented successfully 
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Current State 


Case studies 


- With quantitative evidence of process improvement & concomitant 
performance gains 


- Typically showing total results over time ... often rolled up over multiple 
projects 


Often accompanied by qualitative affirmations 
- Based on experience of those doing the work 


- That process improvement is the major source of change 


But little explicit discussion of competing explanations 
¢ Whether or how they were considered 


- Leading to accusations of spurious correlation 
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Understanding Variation & Causal Thinking 


There almost always is variation 
In product quality, project & organizational performance 


- How processes are enacted 

— The existence of defined processes 

— Adherence/compliance with them 

— & how well the processes are enacted ... 
& the other factors that may effect both process & performance outcomes 


the “goodness” issue 


Looking only at total results 
May mask important differences at the project level ... 
development work actually takes place 


{Per an important statistical literature about possible misinterpretations from 
looking only at the marginal totals as opposed to individual data points} 


where most of the 
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Bases for Comparison 


Organizations vary widely in the measures that they use 


& their use of measurement altogether 


No industry standard measurement criteria yet exist 
Even if they did exist ... organizations in our field are cautious to say the 
least about sharing proprietary information about their business assets 


Organizations differ widely in how they establish processes that satisfy 
the goals of CMMI 
Other factors must be considered in addition to process & performance 


Staff capabilities (skills & expertise), tools & technologies used, product 
domains, precedentedness, interactions with acquirers & SO on 


All of which can & should be the subject of defined processes 
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Our Evolving Research Agenda 


A two-fold approach 


1. Examine performance effects of variation in maturity level & 
capability level 


- For that we need wider variation in process enactment 


- Results most pertinent for organization less far along in their 
improvement journeys 


2. Higher maturity organizations (by definition) have (or should have) 
less variation in what processes they follow 
- But there can be, & sometimes is, variation in how they do it 


- &some variation does remain in their performance outcomes 
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Systems Engineering Effectiveness 


Survey of DoD Program contractors 
- Incollaboration with NDIA Systems Engineering Division 


¢ Under strict conditions of non disclosure 


Measures of: 
- Earned value, other performance criteria, project & product context 


- Process capability questions about existence, use & quality of interim 
process work products 


- Consistency with appraised maturity levels 


Results currently expected in 1§' quarter of CY2007 
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Performance Measurement Associated with 
DoD Program Monitoring 


Independent judgments about programs’ adherence to processes 
consistent with appraised maturity of parent organizations 


Program monitoring by DCMA ... along with surveys of performance 
outcomes 


Currently considering similar analyses with ~20-30 programs for which 
monitoring appraisals have been done by SEI... & possibly others of 
comparable quality 


Possibly in concert with a “SCAMPI+” performance module 
Appraising project/program performance 


Linked directly with process appraisal results 
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The Performance Benchmarking Consortium 
(PBC) 


Members with existing quantitative measures & experience using them 
Working together to create (& then enhance) a common product suite 
Measures & a repository infrastructure to 
Facilitate benchmarking comparisons 
Provide CMMI-based measurement guidance 


Supported by appropriate trailing activities 


Come to Mark Kasunic’s presentation for much more detail 
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Enterprise Wide 


Performance Benchmarking 


Software Engineering Institute 


Carnegie Mellon 
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Overview, 


Work with larger enterprises 


- Comprised of multiple projects, contractual programs, & other 
constituent organizational units 


- That already have &/or are evolving common measures of performance 


- Along with complimentary measures of process enactment, 
organizational & product attributes 


An important way to provide: 
- More generalizable, comparative analyses explaining variation 
- In both successful process enactment & the performance outcomes 
that the processes are meant to achieve 


Especially important /n /ieu of shared measures that enable wider state 
of the practice & benchmarking analyses 
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Overview, 


Participating organizations bring: 


- Validated data that are actually used to inform management & technical 
decisions 


- Personnel and financial resources to support the analyses 
Work now in progress with three selected organizations 
- All want to focus on factors that affect that performance 
- Also prefer to address their own more concrete “how to” questions 


- Augmented with survey or other mining of “tribal knowledge” about 
current variations in process enactment that may account for exemplary 
outliers 


- Initial focus on high maturity organizations within their larger enterprises 
- All three do have somewhat wider capability variation though 
— & BAE has wider maturity level variation 
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Overview, 


Current Participants 
- Motorola Software Group 
- Lockheed Martin Integrated Systems & Solutions 
- BAE 


Work also provides useful how-to guidance for others elsewhere 


- With enough detail to give others a good sense of the participating 
organizations’ processes without divulging anything proprietary 


Work in progress 
- Only a few example results shown here 


- More to follow next year’s conference & in the interim 
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Similar Work, 


Similar studies exist ... Mostly based on the SW-CMMI 


Clark, B. The Effects of Software Process Maturity on Software Development 
Effort. PhD Thesis, University of Southern California, April 1997. 


Deephouse, C.; Goldenson, D.; Kellner, M.; & Mukhopadhyay, T. “The Effects of 
Software Processes on Meeting Targets and Quality,” 710-719. Proceedings of 
the Hawaiian International Conference on Systems Sciences. Wailea, HI, Jan. 3- 
6, 1995. Los Alamitos, CA: IEEE Computer Society Press, 1995. 


K. El Emam, A. Birk (2000). ‘Validating the ISO/IEC measure of software 
development process capability’. Journal of Systems and Software 4(2), 113- 
133: 


El Emam, K. & Goldenson, D. “An Empirical Review of Software Process 
Assessments.” Advances in Computers 53 (2000): 319-423. 


Goldenson, D. & Herbsleb, J. After the Appraisal: A Systematic Survey of 
Process Improvement, Its Benefits, and Factors that Influence Success 
(CMU/SEI-95-TR-009, ADA 302225). 
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Similar Work, 


Goldenson, D.; El Emam, K.; Herbsleb, J.; & Deephouse, C. “Empirical Studies 
of Software Process Assessment Methods.” in K. El Emam and N. H. Madhavji 
(eds.): Elements of Software Process Assessment and Improvement. Los 
Alamitos, CA: IEEE Computer Society Press, 1999. 


Harter, D. E.; Krishnan, M. S.; & Slaughter, S. A. “Effects of Process Maturity on 
Quality, Cycle Time, and Effort in Software Product Development.” Management 
Science 46, 4 (April 2000): 451-466. 


Jung, H. & Goldenson, D. CMM-Based Process Improvement and Schedule 
Deviation in Software Maintenance (CMU/SEI-2003-TN-015, ADA 416418). 


Krishnan, M.S. & Kellner, M. |. “Measuring Process Consistency: Implications for 
Reducing Software Defects.” /EEE Transactions on Software Engineering 25, 6 
(November/December 1999): 800-815. 


Lawlis, P.; Flowe, R.; & Thordahl, J. “A Correlational Study of the CMM and 
Software Development Performance.” CrossTalk 8, 9 (September 1995): 21-25. 


Paulk, M.C. An Empirical Study of Process Discipline and Software Quality. 
Ph.D. Dissertation, University of Pittsburgh, 2005. 


<http://etd.library.pitt.edu/ET D/available/etd-070 82004-15591 7> 
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Higher Maturity Comparisons, 


Focus on finer grained how-to measures 


- Either already existing or collected as needed ... perhaps with custom 
survey data 


- As part of QPM, OPP, CAR & OID activities 


CAR typically ... 
- Looks for plausible explanations of special causes of variation 
— What’s different about the exemplary & least successful projects 
— Both process & other factors 
— Where success is measured by performance outcomes 


- Then defines or redefines processes, trains & implements them & 
monitors the performance effects of the changes in process 
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Higher Maturity Comparisons, 


Proactive, statistically based CAR 
Uses data from existing & enhanced measurement repositories 


— To examine as-is statistical relationships 


Since there is less process variation in higher maturity organizations 
Probably makes more sense to look first for repeatable patterns between 


performance & other contextual factors 
Then identify, define, refine & evaluate the effects of new or changed 


processes 
Where appropriate, use quasi experimental OID processes to 
Pilot, incrementally deploy & modify the new processes as necessary 


based on measured performance results 
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More “High Maturity” Measurement Guidance: OID 


It is reasonable to pilot first with accommodating projects 


To refine new processes before wider deployment 


But need some degree of experimental control, e.g., 


— Matching or paired comparisons 
— To attribute change to the process intervention as opposed to 
Spurious correlation due to other factors 


— Then track & revise aS necessary 
Incremental deployment yields more confidence in cause & effect 


- As opposed to “training pants” learning effects 
But need baselines of as-is state prior to process intervention 
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More “High Maturity” Measurement Guidance: 
CAR, 


“Cross sectional” statistical studies are less vulnerable to accusations 
of learning effects due simply to staff maturation over time 


- But we do need process variation to separate process effects from 
competing explanations of differences in performance 
Historical data in measurement repositories & other existing 
measurement infrastructure 
- Make it easier for higher maturity organizations to make comparisons of 
otherwise comparable circumstances 
The R in CAR still needs to be compared with an otherwise like 
baseline 


- But that’s possible when incremental changes are made with the same 
people, same product & project characteristics 


- Especially after similar new projects apply the same new processes 
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More “High Maturity” Measurement Guidance: 
CAR, 


One can pool cases over time to increase variation in process 
enactment 


- It’s still necessary to control for staff experience & learning effects 


— Even though experience per se doesn't always explain all that 
much {as per the early research on programmer productivity} 


— Still, we need better measures of staff skill, e.g., design, code, 
domain knowledge 


- With enough cases, the mix of contextual variables hopefully can be 
controlled statistically ... not just experimentally 
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Cost of Quality at Motorola Software Group (MSG) 


Cost of Quality (COQ) at MSG includes effort spent on... 
Review / Inspection 
Test development & execution 


Quality auditing, training, other process Improvement & problem 
prevention 


Cost of Poor Quality (COPQ) includes... 


Rework & related failure correction throughout the life cycle 


Both expressed as percentages of overall effort soent for product 
development 


Results presented here examine test development & execution 


Initial results from 2002 drill-downs ... N = 46 projects 
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Variation in Cost of Quality for Test Appraisal 


Results from MSG 
China Center 
- N=46 
¢ Actual values of 
data distribution 
(not shown here) 
are quite good by 
our sense of 
“industry 
standards” 
- Still, there are 
variations that 
MSG wishes to 
reduce further 


/ 


LL, 
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Differences by Domain 


DSP/Multimedia projects Boxpl ot of GoRTest by Dnain Gt egory 
have higher COQ-test 


Most Assembly projects 
fall here 
Some porting & 
optimization-oriented 
development projects, 
with high performance 
requirements 
& some product-oriented 
projects 

Embedded vary more 


Ns: 


10 DSP Multimedia 
29 Embedded 
2 Telecom; 5 Tools 
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Differences by Motorola “End Gates” 


Projects ending at Boxpl ot of GOQTest by Ending Gite 
System Test often 


have much higher 


COQ-Test 

Ns: 
5 Code & Unit 
Test (CUT) 


20 Component 
Test & Feature 
Integration Test 
(FIT) 


21 System Test 
(FT) 
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Differences by Coding Language 


Major effects on test Be apices ne ree 
effort & COQ-Test Me est by Coding Language 


Projects coded in 
assembly 
significantly more 
costly to test 


Ns: 


COQ Test 


10 Assembly 


26 Non assembly 


Assenhi y Non- Assenhl y 
Gadi ng Language 
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Explaining the Variation 


Factors that vary jointly with COQ-test - MANOVA Adjusted R? = .67 


Motorola end gates (Code & unit test; feature & component test; or 
system test) 


Delta Code Size in KLOC 
Domain (Multimedia, embedded, tools & telecom) 
Coding language (assembly versus non-assembly) 


Base Code size in thousand assembly-equivalent lines of code 
(KAELOC) 


Other factors examined — weak relationships, not significant 
Project Lifecycle (full or partial) 
In Process Faults (IPF) ... Post Release Defects (PRD) 


COQ for review/inspection ... Total Document Size (pages) 
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Some Actions Taken 


1. Encourage test automation 
- Especially for product-oriented projects 


2. Develop organizational integration & system test guidelines 
- To reduce test development effort 


3. Encourage project test case reuse & automation 
- Especially for long term projects 


4. Enhance analysis of escaped defects 
- Develop causal analysis guidelines 
- Introduce causal analysis methods such as ODC 


5. Optimize regression test strategy 
- Introduce fault prediction tool 


6. Better sharing of practices & lessons learned among projects. 
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Effects of Process Change 


Improvement 
actions reduced 
COQ-test cost 


Ns: 
56 in 2002 
75 in 2004-2005 
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Phase Containment Effectiveness & In Process 
Faults at Motorola Software Group (MSG) 


Phase Containment Effectiveness 
The proportion of faults that are found at the first check point after they 
were introduced 


— Classified by root cause analysis as errors ... 
that escape detection until later 


— Examples shown here are for coding only 
Updated incrementally by phase to monitor & control pertinent processes 


as opposed to defects 


In-Process Faults 
Number of faults found before completion of the project’s final phase prior 


to release Customer Satisfaction 


Related MSG performance measures (not shown here) include ... 
First Estimation Accuracy, Cycle Time Reduction Rate, Post-Release 
Defect Rate & Customer Satisfaction 
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Variation in Phase Containment Effectiveness 


Results from MSG 
China Center 
- N= 87; pooled 


Actual values of 
data distribution 
(not shown here) 
are quite good by 
our sense of 
“industry 
standards” 

Still, there are 
variations that 
MSG wishes to 
reduce further 
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Phase Containment Effectiveness by MSG Center 


Boxpl ot of PCE Code [% by Center 


Results from 10 MSG 
Centers 


- N= 487 


- All Centers do 
well by our sense 
of “industry 
standards” 


- Again, there are 
variations that 
MSG wishes to 
reduce further 


Cent er 
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Phase Containment Effectiveness by Domain 


Better phase Boxpl ot of PCE Code [% by Domain 
containment for 
Embedded projects 


Where China 
Center has more 
experience 


Projects to develop 
Software Tools are 
newer work for the 
Center 


Ns: 


66 Embedded 


15 Software Tools Dorrai n 
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Do Other Factors Affect Code Phase 
Containment Effectiveness? 


Substantial differences among product lines 


Center does particularly well with multimedia projects — where they are very 
experienced 


Yet, team experience has no independent overall effect 
Possibly implying adequate process definition & training 
No significant differences by project category 
Customer completes some of the lifecycle 
Normal Development 
DMS 
Porting 
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Variation in In-Process Faults 


Results from MSG 
China Center 


N = 87; pooled from 
2004 through 2006 


Actual values of 
data distribution 
(not shown here) 
are quite good by 
our sense of 
“industry standards” 


Still, there are 
variations that MSG 
wishes to reduce 
further 
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In-Process Faults by Team Experience 


Team experience does Boxpl ot of |PF_1 by Team Experi ence 
affect insertion of faults 


- Implying need for 
improved staffing, 
training, coding 
standards or other 
improved processes 


| PF_1 


Ns: 
- 6 High x 
- 6/7 Medium 
- 10 Low 


H gh Low Medi um 
Team Experi ence 
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In-Process Faults by Domain 


Marginally more faults in Boxpl ot of |PF_1 by Domain 
Embedded than Software 
Tools development projects | x 


- Could be a function of 
greater complexity, imply 
need for improvement on 
engineering processes, 
or simply be statistical 
noise 


- 66 Embedded | +. 


¢ 15 Software Tools 


eBe 


aK OB 


Ns: 


Domai n 
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Synopsis: Lockheed Martin Integrated Systems 
& Solutions 


Performance Measures 
Unit Software Costs 
Find & fix costs 
Overhead rate 
Productivity (for equivalent lines of new code) 
Award fees 
Processes instituted include 
Architecture based design 
scenario based testing 
Design adequacy assessments 
Quality awareness throughout the lifecycle 
Early defect detection through inspections & defect causal analysis 
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Amassing the Necessary Evidence: 
Some Related Examples 


1. Independent measures of process compliance/adherence 
2. CAR at Northrop Grumman IT 


3. Project performance at Warner Robins Air Logistics Center 


lf short on time then punt > | 
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Measuring Process Compliance & Adherence 


To do broadly-based comparisons 


Must already have (or begin to collect) independent measures of 
process capability 


Available from 
CMMI class C appraisals 
Monitor & control activities 


QA tracking of existence, timeliness & quality of interim process work 
products: 


Questionnaires, e.g., in existing SW-CMM based studies 
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Measuring & Tracking Process Adherence 


Work Products Completion 
100% = 
90% 
inte —@— Early Planning 
60% —m— PP 
50% —&— PMC 
40% - —»<— Engineering 
— —*«— Support 
20% its 
10% 
0% T T l 
M 1 M1.5 M2 M2.5 M3 M3.5 


information Services, Inc. 
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CAR at Northrop Grumman IT 


An example of careful CAR 


- Albeit with one large project 


Quasi-experimental control of change over time, 
- Similar difficulty develooment tasks 
- No other known differences over time 
- A step jump after process intervention 


— Unlikely had the process change not taken place 
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CAR at Maturity Level 5: Northrop Grumman IT, 


6.6 


2.1 


Defect Density (Defects/KLOC) 
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CAR at Maturity Level 5: Northrop Grumman IT, 


Similar results for cost & schedule variance 
ROI: ~13:1 
Hours saved fixing had defects not been avoided 


Hours invested on team training & conducting CAR Cycles 
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Project performance at Warner Robins Air 
Logistics Center 


Comparisons of variation in performance over time 


No explicit measurement of variation in process adherence by project 


But the process intervention clearly was the only thing that changed 
other than marginally 


Small, short term maintenance projects 


Almost entirely adding new capabilities / functionality 
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Change in Percent Cost Variance 


1-Before CMMI ML5 2-After CMMI ML5 


Frequency 


%Cost Variance 
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Change in Percent Schedule Variance 


-400 -300 -200 -100 0 100 200 


1-Before CMMI ML5 2-After CMMI ML5 


Frequency 


-400 -300 -200 -100 0 100 200 
%Schedule Variance 
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Today’s Talk 


Setting the stage 

Our approach 

Current & Proposed Studies 

Enterprise wide performance benchmarking 
Example results 


How can you help? << 
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How Can You Help? 
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Participate in Our Work & Contribute Your Own 


We are always on the lookout for serious collaborators for: 
Enterprise Wide Performance Benchmarking studies 
The other work described today 
Any other ideas you may wish to pursue 


Share your experiences with us & the wider systems& software community 
Submit a brief article for DoD Software Tech News, March 2007 Issue 
— Special issue on performance results of CMMI ... 31 January deadline 


— Quantitative case studies welcome ... comparisons of variations among 
projects/programs preferred 


Also editing an issue (perhaps issues) of Software Process Improvement 
and Practice later in the year 


Please see me here in Denver about these & other opportunities 
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Thank You for Your Attention! 


Dennis R. Goldenson 


dq@sei.cmu.edu 


Software Engineering Institute 
Carnegie Mellon University 
Pittsburgh, PA 15213-3890 

USA 
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CMMI for Acquisition: Introducing the 
Acquisition Constellation 


CMMI Technology Conference 
November 2006 


Ms. Kristen Baldwin 
OUSD(AT&L)/Systems and Software Engineering 


Interest in Acquisition Process 
Improvement 


¢ Congress directed the Department to establish software 
acquisition process improvement 

¢ OSD led the development of the CMMI-Acquisition 
Module (CMMI-AM) 

e Services and Agencies tailoring CMMI and CMMI-AM for 
use in acquisition activities 

¢ General Motors and the SEI publish Adapting CMMI for 
Acquisition Organizations: A Preliminary Report 

¢ CMMI Steering Group with DoD Sponsorship, 
established the CMMI-Acquisition project 


New CMMI Architecture 
Concept of Constellations 


¢ Constellation: “A collection of components that 
are used to construct models, training materials 
and appraisal materials in an area of interest” 
— Development Constellation 
— Acquisition Constellation 
— Services Constellation 
e Each constellation adds “additions” to the CMMI 
Foundation Model, which includes core 
components 


¢ The model for the Acquisition Constellation will 
be called CMMI for Acquisition, CMMI-ACQ 


CMMI-ACQ 
Development Strategy 


¢ General Motors and the SEI developed the initial CMMI-ACOQ model 
— Considered CMMI Acquisition Module (CMMI-AM) in its development 


— Involved industry and government agencies in developing and reviewing the 
initial draft model 


— Special report at http:/Awww.sei.cmu.edu/cmmi/models/initial-cmmi-acq- 
release.html 


¢ Additional requirements solicited by piloting 
— Assures Service/Agency/Industry acquirer needs are addressed 
— Enhances acquiring organization understanding of process improvement 


¢ CMMI-ACQ Model Team will propose improvements to the initial model 
based upon findings from the pilots 


¢ CMMI-ACOQ Advisory Board will vet changes 
— Key reviewers identified for stakeholder review 
e Steering Group will endorse final update 


*The initial CMMI-ACQ 
Advisory Board model Requirements and 
OUSD (AT&L) Design was developed with 
Services (3) CMMI-AM, and the GM ACQ 
MDA project, using the CMMI v1.2 
DCMA model as the core 
GAO 
GM 
NDIA 


Model Team 


- CMMI Architect 
- SEI 
Initial CMMI-ACQ (9 $0} 


- Acquisition 
Experts 
(Services, 


DAU, Industry) | (Requirements/Design*) | (Pilot/Review/Revise, | (Pilot/Revise) | 


Making changes to be 
consistent with 1.2) 


Stakeholder Community 


Initial CMMI-ACQ 


¢ GM/SEI Joint Project Goal: Develop an Acquisition Model that 
conforms with the CMMI architecture 


— Based on the CMMI Model Foundation 


¢ 16 process areas that cover project management, organizational and 
support process areas; the generic goals and practices; and glossary 


— Source models included 
¢ CMMI Acquisition Module (CMMI-AM) 
e Software Acquisition Capability Maturity Model (SA-CMM). 


— Incorporates attempts by several acquisition organizations to adapt the 
CMMI for Development (CMMI-DEV) for an acquisition organization 


¢ Best practices contained in the initial draft CMMI-ACQ went through 
an extensive review process 


— More than 1,000 comments and change requests from acquisition 
organizations as well as suppliers 


— Over 60 reviewers from 24 industry and government organizations 


Initial CMMI-ACQ Acquisition 
Process Area Additions 


Core 
Foundation 
Model 


Process Management 
Project Management 
Support 


Specific Goals in CMMI-ACQ PA 
Additions 


¢ Acquisition Requirements Development 
— Develop Customer Requirements 
— Develop Contractual Requirements 
— Analyze and Validate Requirements 
¢ Acquisition Verification 
— Prepare for Verification 
— Verify Selected Work Products 
¢ Solicitation and Supplier Agreement Development 
— Prepare for Solicitation and Supplier Agreement Development 
— Select Suppliers 
— Establish Supplier Agreement 


Specific Goals in CMMI-ACQ PA 
Additions, continued 


e Acquisition Management 

— Manage Supplier Agreements 

— Satisfy Supplier Agreements 
e Acquisition Technical Solution 

— Develop Technical Constraints 

— Analyze and Verify Technical Solution 
e Acquisition Validation 

— Prepare for Validation 

— Validate Products or Services 


CMMI-ACQ Pilots 


¢ CMMI planned/proposed pilot efforts for FYO7 
— Australian DMO (complete) 
— General Motors and Hewlett Packard (in progress) 
— Army — Picatinny Arsenal (complete) 
— USAF — Space & Missile Center; Electronic Systems Command 
— DHS — “US Visit” Program 
— NAVAIR — (thd) 
— Missile Defense Agency (tbd) 
— HUD - (thd) 
¢ UK MoD interest being explored 
¢ GAO and others are piloting as opportunities arise 


Pilot Details 


GAO Review/Agency input 
— 38 CRs already submitted 
SCAMPI-B with GM and Hewlett Packard 


— Conducted at GM site, 2-6 October, to identify strengths, weaknesses, 
and needed improvements 


— Generated 19 CRs, some overall issues and some specific technical 
issues with the special report 


— SCAMPI-C conducted at HP site, 6-10 November 


Australia Defense Material Organization 
— Proposed reorganization for the model 
Army, Picatinny 


— Review conducted with engineering group that provides acquisition 
support to Program Offices 


— Generated several CRs 


CMMI-ACQ Governance: 
Advisory Board 


¢ Includes representatives from major acquisition 
community stakeholders 
— DoD (OUSD (AT&L) and Services) 
— Other Government Agencies (GAO, DCMA) 
— Commercial and Defense Industry 
¢ Ensures CMMI-ACO considers stakeholder needs 
¢ Coordinates with CMMI Steering Group to ensure 
consistency among constellations 


¢ Reviews pilot results and categories of change requests 
to set the criteria for changes to the initial draft CMMI- 
ACQ 

e Reviews and comments on products from the CMMI-ACQ 
model team 


CMMI-ACQ Development: 
Model and Training Teams 


e Model Team 


— Makes changes to the GM/SEI initial draft CMMI-ACQ 
baseline according to criteria defined by the Advisory 
Board 


— Dispositions CRs generated from the pilots against the 
baseline and approved by the Advisory Board 


— Raises overarching issues to the Advisory Board 
— Submits model to the Advisory Board for review 
¢ Training Team 
— Develops CMMI-ACQ-specific training material from 
model 


— Submits training material to the Advisory Board for 
review 


Sep - Nov 2006 


Dec 2006 - Feb 2007 


March 2007 


April 2007 
April 30, 2007 


Schedule 


Conduct pilots with initial draft 
CMMI-ACOQ baseline 

Update baseline based upon 
pilot findings and submitted 
change requests 

Develop training 

CMMI-ACOQ available for 
stakeholder review 
CMMI-ACOQO available for QA 
Release CMMI-ACQ Product 
Suite (Model, Method, Training) 


Summary 


e An integrated approach to developing acquisition process 
improvement 
— Industry involvement 
— Government-wide needs 
— Builds on existing process improvement efforts 
e Advisory Board and pilots will shape the final outcome 
¢ This is an opportunity to transform acquirer and supplier 
relationships and shape the future of process 
improvement 
¢ For more information contact Karen Richter, IDA, 
krichter@ida.org, or Mike Phillips, dmp@sei.cmu.edu 
¢ For a copy of the GM/SEI Special Report go to 
http://www.sei.cmu.edu/publications/documents/06.report 
s/06sr005.html 
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Software Process for 
Systems Engineering 


Tim Chick (a.k.a. Jeff Schwalb) 
Naval Air Systems Command 
James McHale 


Software Engineering Institute 


Carnegie Mellon 
Software Engineering Institute 


Topics 

Need 

NAVAIR/SEI Collaboration 
Approach 

Team 

Research Challenges 
Conduct of Pilot Project 


What’s Next? 
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Need 


TSP is being used with great results on software teams. 


See CMU/SEI-2005-SR-012, CMU/SEI-2003-TR-014, and 
CMU/SEI-2000-TR-015. 


There is growing interest in applying TSP to other domains. 
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NAVAIRI/SEI Collaboration 


NAVAIR already has a great track record with TSP: 
¢e ROI demonstrated on software projects 
e other teams (SE) requested training and launch 
Support 


SEI is also receiving additional requests to apply TSP to 
non-software settings. 


Solving software problems becomes increasingly difficult 
without addressing systems engineering and acquisition 
iSSues. 


—_— Carnegie Mellon 
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AV-8B TSP/CMMI Experience 


AV-8B is a NAVAIR System 
Support Activity (SSA). 


= eo 
They integrate new features into the |. tape 


Marine Harrier aircraft. 


to go from CMMI Level 1 to CMMI 
Level 4. 


SEI Average 


AV-8B 


Software Engineering Institute 


TSP Results at NAVAIR 


Program Size of Defect Density Cost Savings 
Program (Defects/KSLOC)) | from 
Reduced 
Defects 


$2,177,169 


AV mps' | 443 KSLOC a 
P-3C 383 KSLOC an $1,478,243 
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Approach 


Conduct a series of pilot projects to determine if extending 
TSP practices to Systems Engineering and Acquisition 
Management results in measurable improvement 


Use the results of this work to establish a common process 
for both systems and software engineering across the 
NAVAIR Mission Area Teams (MATS). 
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What is TPi? 


Team Process Integrated 

ea multi-year experiment 

e adapt and extend the training, methods, and tools associated 
with TSP 

e targets selected systems engineering and acquisition teams 
within NAVAIR (i.e. already using TSP successfully for software 
development) 
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CMMI, TSP & PSP Relationship 


7 AAA 


organizational > / , a be 
capability 


le 
| | TR | | US 


TSP - Builds 
quality products 
on cost and 
schedule 


Wvever \vevel Wivever 
lili @griigiiTwit 


PSP - Builds 
individual skill 
and discipline 


Carnegie Mellon 
Software Engineering Institute 


TSP Builds Effective Project Teams 


PSP TSP 


Skill-building 


Personal measures Project goals 
Process discipline Team roles 


Estimating & planning Team process 


Team-building Team-working 


Risk analysis 
Team communication 
Team coordination 


Balanced plan 


Project reporting 


Quality management Project plan Status tracking 


Team 
Members Disciplines 
zs 
Integrated 
Product Teams 
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The TSP Launch Process 


Management: Defines project goals 
Answers team questions 
Team: Establishes team roles 
Defines team goals 


Team: Defines the project strategy and process 
Produces process and support plans 
Makes an overall development plan 


Team: Produces quality plan 
Allocates next phase work to individuals 
Engineers produce detailed personal plans 
Consolidates individual plans into a team plan 


Team: Conducts a project risk assessment 
Assigns risks to engineers to track 


Team: Reviews launch work completed 
Prepares management presentation 
Conducts a launch postmortem 


Team: Presents the plan to management 
Defends the plan to management 


Management: Reacts to the team's plan 
Resolves plan issues with the team 
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The TSP Launch Products 


ai 


Business needs 
Management goals 
Product requirements 


one, 


* Team goals ° Team ¢ Task plan ° Team * Quality ° Risk 

° Conceptual strategy « Schedule roles plan evaluation 
design ° Team Plan ¢ Task plans ¢ Alternate 

¢ Planned products defined ¢ Earned- ¢ Earned- plans 

¢ Size estimates process value Plan value Plan 
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NAVAIR/SEI Team 


NAVAIR 

¢ Tim Chick 

¢ Dennis Linck 

¢ Linda Roush 

° Jeff Schwalb 

¢ Paula Strawser 


SEI 

e Anita Carleton 

e Noopur Davis 

e Watts Humphrey 
e Jim Over 


NAVAIR Systems Engineering Pilots 
e AV-8B Harrier Aircraft 
e E2-C Hawkeye 
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Research Challenges 


As we kicked-off this effort, we realized that there were 
five areas of TSP that specifically had to be addressed 
for SE: 

e Processes 

e Measurement 

e Role Definition 


e Training 


¢ Tool Support 
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Research Challenges - Processes 


Develop prototype processes/scripts for SE 
Develop prototype processes/scripts for ACQ 


based on: 


¢ the DoD 5000 series regulations 
¢ CMMI Acquisition Module 


Used “traditional” TSP launch process 


Cart Mellon 
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Research Challenges —- Measurement - 1 


Schedule and effort measures are essentially 
unchanged. 


Lines of Code/Function Points would not serve 
as relevant size measures for SE/ACQ. 
Formulate size measures for SE and ACQ. 
Examples: 

¢ DOORS objects 

e Requirements 

¢ Verifies 
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Research Challenges — Measurement - 2 


Quality measures in SE 
¢ Define what “quality” means in SE 
e Where in the process do you collect data? 
e What are the derived quality measures (e.g., 
defects/DOORS object?) 
e Establish an initial quality baseline during Build 1 


L/ 
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Research Challenges — Measurement - 3 


What are the quality goals? Examples: 
¢ Goal: Accuracy in the work 
Measure: # of problem reports against requirements 


and test documents 
e Goal: Conformance to standards 


Measure: # of defects in peer reviews; # of defects in 
requirements and test documents, etc... 
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Research Challenges — Role Definition 


Apply four primary roles—planning, process, 
quality, Support 


Assess applicability of remaining roles and 

define additional roles needed for SE and ACQ. 

e Added Requirements Manager 

e Design and Implementation roles were 
combined into one role 


¢ Test Manager role expanded to Flight Test 
Manager and Lab Test Manager 


SJ Carnegie Mellon 
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NAVAZAIR 
Research Challenges — Training - 1 


Currently, our training is geared to software teams 
Our challenges: 


¢ building conviction and discipline in teams that 
don't write software programs 


¢ providing just the right amount of training to 
get a team started 


¢ supplementing with additional training modules as 
needed 
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Research Challenges - Training - 2 


Develop “JIT” training to support SE teams 


Develop Leadership Seminar and Team 
Member Training to focus on: 

¢ providing the fundamentals of TSP 

e launching a team 

¢ maintaining a plan 


Follow-up with additional, “JIT” training, e.g., 
e Inspections 


e Measurement, data analysis, and reporting 


e Checkpoints and Postmortem Analysis 
¢ Tool 


SJ Carnegie Mellon 
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NAVAZAIR 
Research Challenges — Support Tool 


Develop an extensible tool that allows for: 
¢ Defining any process 
¢ Collecting data unobtrusively 


¢ Defining a measurement framework 
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Progress 
SE Pilot Projects Selected (AV-8B and E-2C) 


SE/ACQ Prototype Processes/Scripts 
Developed 


Training Developed 


Prototype Support Tool Developed 


AV-8B Team Trained and Launched 
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Some Early Data 


Launch Sept. 2006 ... Ran like a “normal” launch 

e Two year overall plan 

e Near-term plan is 4 months 

e 475 tasks 

e 12 team members 

e 22,000 task hours 

¢ Gantt Chart didn’t provide visibility into all of the tasks 
that had to be completed 

e Team members engaged in discussions of what the 
work would entail, dependencies, and what “task 
complete” meant 


Issues: 
¢ Level of granularity of the plan 
¢ Defining appropriate roles for SE Projects 
¢ Defining the SE process 
¢ Developing a quality plan 


—_—_ e Me 
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What’s Next? 


Complete NAVAIR pilots 
Expand NAVAIR use as warranted 


Incorporate lessons learned in TSP Program 
Plans 


Evaluate prototype tools and courses for broader 
use 
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Developing Systems Engineering 
Processes from 
Existing Software Processes 


Lee Sheiner 
Jeanne Balsam 
Mark Pellegrini 


Electronic Systems Laboratory 
Georgia Tech Research Institute 
Georgia Institute of Technology 
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Georgia Tech Research Institute (GTRI) Overview 


Georgia || Research 
Tech 


Unit of the Georgia Institute of Technology 

1200+ employees 

70% of research employees hold advanced degrees 
Wide variety of products 


Customers include federal and state government; 
and industry 


Competitively bid projects range greatly in size and 
duration 


More Info: http://www.gtri.gatech.edul/ 


Institute 


Overview 


Transitioning to Systems Engineering 
Configuration Management 

Project Planning 

Peer Reviews 

Summary 


Questions 
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Systems Engineering 


Institute 
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CSS 
Starting Point 


¢ Software developers were using CMM level 3 
development processes 


¢ An increasing number of programs included 
hardware components 


¢ Developers with SW and HW expertise were 
informally carrying some processes to HW 
development 


¢ Hired additional QA/Process Engineer with 
hardware development experience 
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HW Development Process Overview 


Georgia || Research 
Tech | institute 


Beginning the Transition 
¢ Removed “software” from our documented 
processes and procedures where appropriate 


¢ Discovered that many existing procedures mapped 
reasonably well to Systems Engineering 


¢ Identified additional items for project planning and 
peer reviews 


¢ Configuration management did not map well at the 
lower-level details 
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Work Products by Phase - Planning 


ALA LALA 
A lA LA 


Work Products by Phase - 


Requirements 


Work Products by Phase - Design 


Design 
Document 


af 
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Work Products by Phase - 
Implementation 


Circuit Card Assembly Printed Wire Board i 
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Work Products by Phase - 
Integration and Test 


Ct tf 


By 
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CM Issues: What to Control? 


¢ Many new types of configuration items are 
introduced with hardware 


¢ Developers need guidance in what to control - 
important interim Cls are often overlooked 
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CM Issues: New Cls to Control 


¢ Gerbers ¢ Data Sheets 

¢ VHDL ¢ Symbol Libraries 

¢ Netlists ¢ Technical Data Packages 
¢ Schematics ¢ Assembly Drawings 


¢ Cable Drawings — - Text Fixtures / Test Code 


- BOM! Parts List + Master Pattern Drawings 
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CM Issues: Don’t lose Control 


« Remember to control all files associated with HW 
development 


¢ Items being fabricated (internally or externally) must 
be put under configuration control in advance 


¢ Maintain control of physical HW items — which 
version of the card, firmware, embedded SW, and 
wire-mods are on the card 


¢ Work products documented using multiple 
configuration items must be baselined 


¢ Beware of “back-of-the-envelope” and red-line 
issues in fabrication 
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Example Directory Structure 


System 


CCA-1 
SW 
HW 


Assembly 
Datasheets 
Development Folder 
Documents 

Gerbers 

Plans 

PLDs 

Schematics 
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SSCS 
File Types 


Schematics Gerbers 
Sheet 1 Layer 1 


Sheet N Layer N 


Netlist .jpc Test File 
Symbol Library 


Assembly 


Mechanical Outline 
Assembly Drawing 
Master Pattern Drawing 
BOM / Parts List 
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File Types — Continued 


Documents 


Plans 
Requirements CM Plan 
Design Description 
Interface Control Document 
Baseline Documents 

Test Descriptions 

Product Version Description 


Development Folder Data Sheets 
Test scripts 


Project Plan 
HW Development Plan 
Test Plan 


Parts vendor info 
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Project Planning Issues 


¢ Long lead times / critical path 
¢ Facilities and equipment management 
¢ Developmental baselines needed for fabrication 


¢ “Hidden” software items 
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Peer Review Issues 


¢« Consider schedule impact as well as cost- 
effectiveness 


¢ Proper baselining of reviewed materials with 
multiple configuration items 


¢ VHDL (Very High-speed Integrated Circuit Hardware 
Description Language) is software! 
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Peer Review Issues 


¢ Review hardware configuration items such as 
Gerber files, drawing packages (schematics, wiring 
diagrams, assembly drawings, parts lists, 
mechanical part designs, etc.) 


¢ Select reviewers with HW and SW knowledge - be 
aware of software denial 


¢ Review test software needed to exercise hardware 
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Beware 


¢ Software “hiding” inside hardware items 

¢ Tracing hardware requirements to hardware design 
¢ Fabrication from uncontrolled designs 

¢ “Quick fixes” that don’t get documented 

¢ Changing HW configurations 


¢ Software needed for HW testing 
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Acronyms 
¢ BOM - Bill of Materials 


¢ CCA - Circuit Card Assembly 

¢ Cl -— Configuration Items 

¢ CMM - Capability Maturity Model 

¢ CMMI - Capability Maturity Model Integration 

¢ GTRI -— Georgia Tech Research Institute 

¢ HW —- Hardware 

¢ PLD -— Programmable Logic Device 

¢ SW - Software 

¢ VHDL - Very High-speed Integrated Circuit Hardware Description 


Language 


Geo 
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Contact Information 


¢ Lee Sheiner 

¢ lee.sheiner@gtri.gatech.edu 
¢ Jeanne Balsam 

¢ jeanne.balsam@gtri.gatech.edu 
¢ Mark Pellegrini 


¢ mark.pellegrini@gtri.gatech.edu 


¢ More Info about GTRI: 
http:/Mwww.gtri.gatech.edul 
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Lean CMMI 
Maturity Level 3 
Process 


Zia Tufail, Zia@hp.com, 301.233.4228 


Julie Kellum, Julie.Kellum@hp.com, 404,731. 52.63 
Tim Olson-QIC, Tim.Olson@gic-inc.com, 760.804.1405 
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Presentation Objectives @ | 


Describe CMMI compliant online HP process 
Outline Components for a cost effective, lean process 


Describe HP’s improvement objectives. 


Identify problems addressed by the streamlined CMMI L3 HP 
process 


Describe the pEploace that was used to achieve CMMI 
Maturity Level 3. 


ele techniques for defining & implementing an effective, 
lean CMMI L3 process in 8 months 


Present some challenges and lessons learned. 


Answer any questions. 
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HP Improvement Objectives 


HP Approach 


Challenges and Lessons Learne 


Questions and Answers 
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FY05 C&l EAS/Federal Initiatives | 


EAS & Public Sector Initiative 


CMMI L3 Achievement 


Expand Application 
Services 


Increase Pursuit 
Capabilities 


Delivery Excellence 


Profit Improvement 
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CMMI Relation 


Development of EAS HPGM-AS 
methodology, rollout and 
assessment. 


L3 methodology is unifying force for 
consistent standards and practices. 


CMMI allows us to pursue 
Opportunities we would not get 
otherwise 


Better estimation and marketing of 
services 


Standardization and enforcement of 
best practices. 


Disciplining delivery execution and 
preventing margin leakage by 
fostering reuse and minimizing risk 


4 


2004 


A $1.5 Million dollar, 8 month project started to revise Success Progra 
meet CMMI level 3 and improve project controls. Some current and new 
FYO5 Federal Contracts at risk without a CMMI level 3 compliant process. 


Existing 
Methods ¢ Sept CMMI Level 3 Gap Analysis on Current Methodology 
Revised to ° Oct -Nov HPGM-AS Methodology Customizations - define per 
Meet CMMI Sharepoint portals 
Level 3 ° Dec Pilots Start : 
*PM Orientations/Planning Sessions per project 
eCMMI & Methodology Classroom Training 


2005 


Success Program Office (SEPG team) and Pilot Projects start leveraging 
Success Program revised with the new CMMI level 3 compliant process 
Pilots e Jan Envisioning & Maintenance Phase Pilots start 
Underway ¢ Feb Design & Maintenance Phase Pilots 
e Mar Build & Maintenance Phase Pilots 
e April Pilots Continue & CMMI Assessment begins 
e April 29 EAS and Public Sector Achieve CMMI level 3 maturity 
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Components of the HP CMMI L3 Process =: 


Technology 


HM Sharepoint based portals: 
Success Program Office 
Success Manager 
Success Reviews 
MHPGM-AS Roadmap 
RADM Virtualization 


Process 


@ SEI CMMI L3 framework 


HP Global Method Application Services 
(HPGM-AS - methodology) 


i Success Reviews (PPQA audit process) 

Mi Success Manager (Team Collaboration 
process) 

@ Adaptive Assets (Knowledge Capture & 
Reuse process) 


Success Program Office @ | 


EAS HP Global method 
Application services 
(HPGM-AS) 


Success Manager 


Adaptive Assets Process and Tech Asset KB portal/process 
Knowledgebase Pauceionl 


Rapid ADM 
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Delivery Workflow portal/process 
CMMI level 3 compliant 
Phase deliverables & Signoffs 


Team Collaboration web/process 


Pre-Loaded HPGM-AS 
Customer accessible 


Process Review (PPQA) portal/process 


Mentoring Support By Field PMs 


Supports HPGM-AS Configuration Mgt 
Virtualization and Software Configuration 
Management (SCM) services 


Edit View Favorites Tools Help 


sk Q(x) (GD Arsene gyracnes Qh @ S- Sa IM#Saas 


SS @) https://h20224. www2.hp.com/default.aspx 


My ietrectelac Office 
SharePoint Portal Server 2003 


Rapyi ADM 


» Rapid ADM 

» SPO CMMI 

» HPGM-AS 

» SuccessManager 

» Success Reviews 

» AdaptiveAssets 

» Communities 

» Success 
Scorecard 


ctions 

Add Listing 
Create Subarea 
Upload Document 
Change Settings 
Manage Users 
Manage Content 
Manage Portal Site 


SPO CMMI HPGM-AS SuccessManager Success Reviews 


Success Program Office 
Home 


Welcome to the Success Program Office! 


HPGM-AS PROCESS 


AdaptiveAssets 


My Site Site Setting 


Communities Success Scorecard 


a ENIPortal SearchS |v 


Contact: 
Brian Erwin, Julie Kellum 


There are currently no upcoming events. To add a new event, click "Add new event" below, 


Announcements ¥ 


Self-Service Site Creation 7/15/2006 6:13 PM 
by sve-sharepoint 

Self-Service Site Creation has been enabled for this virtual 
server. Go to 
http://h20224.www2.hp.com/_layouts/1033/scsignup.aspx to 
create a new root Web site. 


Submit Success Program Improvement 3/4/2005 10:23 AM 
Requests Here! 
by Julie Kellum 


Please go to this link to submit Success Program 
Improvements https://h20224.www2.hp.com/Lists/SPO% 
20Change%20Requests/Sort%20By%20CRPRid.aspx 


These are reviewed at the weekly Change Control Board (CCB ) 
meetings by SPO... 


Submit SPO Issues Here! 
by Julie Kellum 


Used to track SPO issues, the resolution of some of these may 
result ina SPO CR/PR Success Improvement Request for the 
HPGM-AS process, artifacts, or SPO web sites 
https://h20224.www2.hp. com/Lista/SPO%20Issue% 


Se ee 0 ae 


3/4/2005 10:23 AM 


SPO Links 7 
» Request Success Manager Team Web Here! 


» Submitting Success program Improvements 
© SPO Issue Tracking 


« SPO Quality Policy Document 


» Identifying and Reporting a Security Incident 


» HP Security 
» CMMI Scope 


ps fh2022 aie 


@ Home - HPGM-AS Roadmap R2.6 - Microsoft Internet Explorer provided by Hewlett-Packard Sel x) 


sl 


View Favorites Tools 


File Edit Help 


Gre CE x) 2) GD Psere gpravontes QPrea @ -So L) BGO Ds 


Address €) https: //h20224. www2.hp. com/sites/ProjectsCMMi/SuccessPath_| HPGM-AS _Roadmaps/Roadmap_ v6/default. ASPX 


v E} co Links 


' Home Documents and Lists Create Site Settings Help Up to Success Manager HPGM-AS Roadmaps 4s 


A HPGM-AS Roadmap R2:6 [el[thisste wa | 
\ ) Home 


Modify Shared Page y 
Documents Second Release of HP Global Method for Application Services which is a level 3 CMMI application development 
HPGM-AS ZIPed methodology supporting iterative, waterfall and maintenance lifecycles in HP's Consulting and Intergration (C&I) area 


Templates HPGM-AS ¥ CMMI Links ¥ 


Pictures = SUBMIT Success Program 
1 
Lists poate Here =I 
New Projects « Adaptive ets 
START HERE » Success Reviews 
Lifecycles a « CMMI project webs (currently USPS 
? Pegacs Tapa only - special clearance required) 
ENVISIONING » HPGM-AS Lifecycles Overview 
PHASE 
«= SPO Portal 
PESGn Ener » SPO Lessons Learned Analysis 
BUILD PHASE « HPGM-AS Impact Analysis Tool 


ARTIFACTS Project Planning (PP) ® SPO Configuration Management Plan 
Project Planning ‘raetag comer] = SPO Training Plan 


PE) Decision Analysis & Resolution (DAR eSPO i il Plan (SIP) 
Project © SPO Quality Policy 

Management Tailoring ‘ 

(PMC) « SPO Quality Assurance (QAP) Plan 


. « CMMI Scope & Design Decisions 
retarniae coat Configuration management(CM) 
anagemen' @ Add new link 
Verification Z validation V&V) 
Custest Leed 


Verification and 
Validation (V&V) 


Contacts Y 

Peer Review (V&V) Success Reviews PPA) ey Last Name First Name Role 

Decision Analysis ; / Success Reviewer Galvan Carlos April 18-29 

Resolution (DAR) Measurement & Analysis (MA) Lead CMMI 

Te *~ Assessor 
Tailoring (IPM) Urg — : . 
- i Sa ; : g Erwin Brian Centers of 

SPO Success i Excellance 

Reviews (PPQA) — Organizational training (OT) SPO Managers Manager 9 
€] Done & Local intranet 


start ~ € CG i 3 Microsof,,, ~ [&) 5 Microsof.., fe Training 20.. f HP4 2006 Hy Home -HPG,. QB | |||58 ec ls [)) 10:46PM 
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e Rapid ADM 


ange Settings 
nage Security 
nage Content 
nage Portal Site 


SPO CMMI HPGM-AS SuccessManager AdaptiveAssets Communities 


Success Program Office 
Success Reviews 


Success Review Home Page 


Welcome to the Success Review Home Page! 
This site defines the PPQA Success Review schedule and lists the results of the PPQA based CMMI audits. In 
addition, the templates used in Success Reviews are shown in LINKS section. 


Announcements 


Success Review Schedule 5/8/2006 4:11 AM 


by Julie Kellum 


Latest schedule for all CMMI project and SPO PPQA audits: https://h20224.www2.hp.com/C15/Success% 
20Review%20Scorecard/Lists/Success% 20Review%20Summary/Allltems.aspx 


Auditors Place Results here 8/12/2005 9:42 AM 
by Julie Kellum 

https://h20224.www2.hp.com/C2/SuccessReviews/Document%20Library/Forms/Allltems.aspx 

Success Review R2v6 Process 6/15/2005 6:38 AM 


by Julie Kellum 


PPQA-based process that outlines how audits will be performed to monitor and control HPGM-AS process 
compliance by project's that fall within the CMMI scope area 


https://h20224.www2.hp.com/sites/ProjectsCMMi/SuccessPath_HPGM-AS_Roadmaps/Roadmap_v6/Lists/Process% 
20and%20Product%20Quality%20PPQA%20%20INSHORE/ByActivityId.aspx 


Success Review Findings 3/15/2005 6:02 AM 


by Julie Kellum! _ 


Latest CMMI PPQA Success Review audit findings: 
https://h20224.www2.hp.com/C2/SuccessReviews/Lists/NC%20List/Allltems.aspx 


© Add new announcement 


LiL 


Success Scorecard 


BE} This topic 


Contact: 
Debra Henrichs 


[ Links 


Success Review Reports | 
Success Review Summar 


Archive of Records while 
Audits 


Organizational QA Plan 

Success Review Template 
Organizational QA Plan (t 
Success Review Kickoff M 
Planning (for Intervieww) 
Success Review Report (t 
Success Review Summar 
Success Review Plan (ten 
SPO Audits of Success Re 
BSI - Online Course - "Es 
Sample Questions Check 
Success Program Improv 


© Add new link 
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ardware 
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perating 
ystems 


rchitecture 
elivery Metrics 
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zate Subarea 
ange Settings 
nage Security 
nage Content 


SPO CMMI HPGM-AS Communities Success Scorecard 


fAdaptivenssets|iig This topic |v 


SuccessManager Success Reviews 


Success Program Office 
AdaptiveAssets 


HP leverages the Adaptive Assets KB web to provide reusable design patterns, code and process to Contact: 
projects. It improves quality and cost effectiveness of new solutions being developed. Brian Erwin 


Announcements 


9/1/2004 9:56 AM 


Welcome! 
by Brian Erwin 


Charter 
Our charter is to search for and harvest reusable code and best practices throughout HP. 


Mission 
Our mission is to drive the harvesting and marketing of HP coding and best practice assets. These assets include non-customer specific... 


Asset of the Month 
» AgFirst Front-End Architecture 


Area Contents 


Software HPGM-AS Process 

“NET Technologies Non-CMMI Samples Most Popular Assets 

Deans eee iri i aia « BizTalk Automatic Canned Functoid Builde 

J2EE Non-CMMI Logistics Path Samples Y : 

Mobile HPGM-AS Release Archive 2 Sepacty hive, 

Web Services HPGM-AS Assessments B Bae Denar Pr aenae 

Best Practices HPGM-AS Lessons Learned 

Collaboration HPGM-AS Metrics Premium Assets 

Databases HPGM-AS Training » BizTalk Automatic Canned Functoid Builde 
HPGM-AS Peer Review Analysis » Build Assistance Tool 

Hardware HPGM-AS SPO Status Reports » CryptoModel block 

Desktop HPGM-AS Estimation » Finite State Machine component 


_HPGM-AS Project Dilan Snnrawale ee eee Pe a 
ill 


My Site Site Settings 


Edit View Favorites Tools Help 
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ss |@) https: //h20224.www2.hp.com/C 15/SPO%20CMMI default.aspx 
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HPGM-AS 


Success Program Office 


My Site Site Settings 


SuccessManager Success Reviews AdaptiveAssets Communities Success Scorecard 


HB} this topic 


SPO CMMI 
Success program Office (SPO) web site for managing CMMI intiative Contact: 
Julie Kellum 
HPGM-AS Release Schedule {Primary SPO Links 
Title Release Link Release Date Release CRs SPO Action Items 
R1.5 HPGM-AS R1.5 12/1/2004 n/a SPO Change Requests/Problem Reports 
R2.6 HPGM-AS R2.6 4/18/2005 CLICK Here for list of CRs for this Release (CR/PR) 
R3.0 HPGM-AS TBD 9/30/2005 CLICK Here for list of CRs for this release ———— 
R4.0 HPGM-AS TBD 10/28/2005 CLICK here for list of CRs for this Release © Add new link 


® Add new announcement 


SPO Workbench 
“SjNew Item | ¥Filter 
SPO Id Edit @ Process Area Category SPO Activity LINK Note 
310 i} OT Training Individual Training Plans HP Performance Plans - Contains Training Plans 
330 | OT Training Training Effectiveness Contains quarterly assessment of training feedback and effectiveness an 
Analysis communication of analysis to the SMSC 
300 A OT Training Training Feedback Forms Training feedback form used for eliciting feedback - used in Training 
Effectiveness Analysis - Note that original feedback forms kept on file for 
290 i} OT Training Training Records Contains the SPO Training records on process training completed as well 
links to Learn @HP which provides online training records for all employe 
Learn @ HP is leveraged to track training for skills outside of the HPGM-p 
training requirements 
PPQA Success QA Plan (Success Review) Online Schedule and ‘Status of Success Review audits planned; Used by tl 


370 A 


ill | 


Edit View Favorites Tools Help 
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PSs | http: //na.know.hp.com/teams/easamericas/RapidADM/default.aspx 


Home _ Help 


Up to Enterprise Application Services - Americas 


42) Rapid ADM 


QA Technology 


Velcome to the Rapid ADM! Y 
© Americas Rapid ADM supports development and QA environments by providing 

pid environment provisioning services. A new environment can be built in as little as 
} minutes with the operating system and development tools installed and ready to 

e. 


day, we support: 

erating Systems: Windows 2003 and Red Hat Linux 

jtabases: SOL Server and Oracle 

ols: Visual Studio, Eclipse, QA Center, Rational Suite, BEA Weblogic, and MS Office 


pid ADM is a division of HP C&I EAS, managed by Brian Erwin, and provides both HP 
ernal and external application teams with tools and consulting services. 


you are interested in getting started using Rapid ADM to provide a dynamic dev/test 
vironment for your team, start by completing the qualification worksheet below and 
ad the service brief for an overview of our offerings. Then, do one of the following: 


r EAS and ENI teams, click on the following link: 


(p://16.114.12.104/RADMPortal/WebPages/SPO/GettingStarted.aspx? 
ferrer=https://h20224.www2.hp.com/C10/RADM/default.aspx 


r other HP teams, fill out a Service Quote Request form and follow the enclosed 
‘ections to get off the ground quickly with your own VMs! 


urs, 


Modify This Workspace v 


Links 
= GDIS VM Service Offerings 
= HPS Knowledge Network 


= Podcasts (RADM is Episode #6) 


Related Sites 


= Test and Development Team Site 


« Virtual Desktop System 
= Tycoon 


= TSG ESS VMWare from HP - Tools 


® Virtual Iron 


Online Videos 
Name 


Introduction to Tetrix 


Tetrix Administrative Features 


Marketing 
Name 


Rapid ADM External Overview 
Rapid ADM in HP-IT 

Rapid ADM Sales Quick Start 
Rapid ADM Value Proposition 
RapidADMSolutionFlyer 

Tetrix Animated Slideshow 


Tetrix Overview v3 
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Questions and Answers 
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Success Program Vision 


Using a sales and delivery integrated approach, leverage 
existing, as well as grow new process and technical 
Adaptive Assets to : 
Increase ability to compete in Federal bids through CMMI Level 3 
compliance 
Win more business through improved differentiation 
IBM differentiation point 
Increase Success of Projects 
Synergistic Teams 
Referencable Clients 
On Time 
In Budget/Profitable 
Effective HP Global Methods Applied 


November 28, 2006 15 


invent 


Success Program = What & Why 


Collection of specific process and technical assets used in pre-sales for 
differentiation and in delivery to increase project Success 


CMMM Maturity Level 3 compliant to meet Federal bid requirements 
Externally accessible outside HP Firewall by delivery teams 


Major Areas: 
Success Path HPGM-AS 
Success Manager 
Success Reviews 
Adaptive Assets KB 
Communities & Forums 


Best of breed IP from HP 
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HP Overview 


HP Improvement Objectives 


Challenges and Lessons Learne 


Questions and Answers 
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HP Approach O) 


Leverage existing HP processes and templates: 
Success Program 
HP India (CMMI Maturity Level 5) 


HP Global Methods for Application Services (HPGM- 
AS) 


HPGM for Project Management (HPGM-PM) 


Deploy Tailored SEI IDEALS” Model and reuse 
existing processes to be CMMI compliant. 


Leverage Best of breed IP from HP 
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SEI IDEALS” Model © | 


invent 
Leveraging 
SM 
SEI IDEAL?” Model for oS 
Process Improvement es Document 
Organizational & Analyze 
Approach Lessons 
a. Define 
Processes 
& Measures 
Stimulus for Set Context Establish b. Plan & Acting 
Improvement & Establish {Improvement Execute 
Sponsorship | Infrastructure Pilots 


c. Plan, 
Execute 


ar aie Appraise & & Track 
Initiating Characterize Installation 

Current 
Practice 

Develop Establish Proces 

Recommendations Action Teams & 
& Document Plan Actions 
Diagnosing Phase Results / set Strategy 


& Priorities 


¢ SM IDEAL is a service mark of Carnegie Mellon University; Slide Adapted from SEI 


Tailored SEI IDEALS” Model @ | 


invent 


Initiating: Sponsor sold idea to HP senior management. 


Diagnosing: HP India performed a mini-appraisal (e.g., 
Class B). 


Establishing: CMMI Consultant established high-level 
plan and HP established a CMMI Team. 


Acting: Architected HP processes to be CMMI 
compliant in 2 months; Skipped piloting; Trained and 
implemented experienced projects. 


Success: Performed independent SCAMPI A. 
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HP Success Program Is a very “lean” CMMI 
compliant process (about 25% of the size of 
the HP India process). 


The process is completely online, and uses 
Microsoft SharePoint. 


HP’s process Is only ~25web pages in size. 


HP incorporated best practices in process 
definition 


e.g., “Defining Short, Usable Processes and Procedures”, 
CrossTalk, Olson, Timothy G., June 2006. 
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HPGM-AS CMMIL3 Methodology 


2 Paths depending on project 


. 2 
profile ar 


Lifecycle 
Phases — 
HPGM-AS 


Large HPGM-AS CMMI 
Implementation 


Project Team 


c) 
Small HPGM-AS CMMI ut 
Implementation 
Excluded are projects / {Braject Wgt Control (PMCs 
ee to Use ie Va fee 
customer's a ev ject \/ :+ / 
methodology & staff aug | sumer 
No existing projects asked \ F 
to convert — eu only __Yatiicaton A valle SY CMiTest Lead 


= i ms 


Zé 


age and Small Project 


implementations differ oo), /_ (aa, Success Reviewer 
“onuse of Success Em) mn) > 
ms, OPD.OPR 
Reviews and —— / 


Organizational training (OT) SOReaee 


Deliverables Required 
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HPGM-AS Artifact Comparison @ | 
eatile eal 


Issues List & Change Req/PR List) (Issues List & Change Req/PR List) 


Mini-Spec Bus Reg Spec (BRS) 
(Contains sections from all of these ») | Systems Reg Spec (SRS) 
Systems Architecture (SA) 
Detail Design (DD) 
V&V /Test Plan 
Req Traceability Matrix (RTM) 
Peer Review Log/Checklist Peer Review Log/Checklist 
Mini-Spec Project plan, SRS, SA, V&V Plan 
CM Audit Checklists CM Audit Checklists 
Success Manger, Product Success Manager, Code, Product 
Milestone Review Phase Checklists Milestone Review Phase Checklists 
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Challenges 


CMMI Resources were 
reallocated to the new 
high priorities. 


Resistance to change 
when people have gone 
through several changes 
without tangible results. 


Aggressive schedule 


Unexpected events such 
as team health and 
organizational changes 
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Lessons Learned 


¢ It takes time, senior management support, and mentoring to 
change behavior 


¢ Traditional Classroom training not enough — critical to have a 
local resource to mentor team on methodology/CMMi activities 
and artifacts 


e Success Reviews critical for monitoring and mentoring on 
process use 


¢ Critical to use centralized team collaboration tool for process 
support (i.e. PPQA, Peer Reviews, Lessons Learned, Issue 
Tracking/Project Change control, et 


= 
SS tSa<- 
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Lessons Learned 
Spend ample time in planning phase to lock down the 
scope of the project. 


Obtain and maintain executive sponsorship to keep driving 
the project on schedule. 


Define Configuration Management early in the process. 
Follow the Configuration Management process. 


There is no substitute for one-on-one mentoring for the late 
adopters. 


Ensure core team understands the entire process. 
Ensure the process is lean and relevant. 


| 


Tl 


| 
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Engage stakeholders throughout the process. 


Third party consultants can provide objective 
feedback. 


Build a diverse team and leverage diversity. 
Ensure CMMI expert resources are on the team. 


CMMI really does require continuous process 
improvement. 
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| What Went Well — 


Teams reap benefits of Peer Reviews, and 
Milestone Reviews 


Pilot projects implement a consistent delivery 
approach 


Improved metrics to track trends across projects 
Integration of virtualization through CM process 


Increased value to customer through improved 
communications thru SPO: 


Success Manager Team Webs 
Status Reporting 
Automated Issues and Change Request tracking 
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What Went Well 


Expanded Rollout of Success program to ~300 


Implemented new Success Program processes: 
HPGM-AS Bid Review (Pre-sales) 


HPGM-AS Quickstarts (Pre-Delivery, 1 week) = tailoring 
of the process by SEPG with PM and SA 


Implemented scalable training to replace 
classroom training: 
PM CMMI Training Track (14 webinars) 
Engineering CMMI Training Track (9 webinars) 


Implemented Success Reviews/PPQA based on 
peer PM resources - not separate SEPG PPQA 
Staff 


Totally integrated into existing funding 
Npembe 2 JOG FL OImMmo 
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FASTrack* to CMMI® 


CMMI® 


Technology Conference & Users Group 
November 13-16, 2006 
Denver Colorado 


Presented by Al Florence 
The MITRE Corporation 


The author’s affiliation with The MITRE Corporation is provided for identification purposes only, and is not intended to 
convey or imply MITRE's concurrence with, or support for, the positions, opinions or view points expressed by this author. 


CMM is a registered trade mark of the Software Engineering Institute The MITRE Corporation 1 
* FASTrack is coined by Al Florence for process improvement 


/ Agenda 


Impediments to Process Improvement (PI) 
@ Real Examples of Impediments 

@ FASTrack to Process Improvement 

@ Real Examples of FAS Track 

@ Summary 

@ References 

@ ?s 

Contact Information 
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What are the Impediments to PI? (1 of 2) 


@ When improving processes in organizations the following 
often occurs: 

> Process Improvement (PI) is done for process sake 

¢» Needs to be done for business sake 

> Management wants immediate feedback 

> Management wants Level 3 in six months 

> Staff fails to “buy-in” to Pl 

> PI efforts linger forever, are ignored, fail, or are cancelled 

> Recidivism sets in 


@ Management is not aware of effort required: 
> Fails to provide adequate funding 

> Fails to provide adequate schedule 

> Fails to provide adequate resources 

> Fails to provide adequate PI staff/skills 
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What are the Impediments to PI? (2 of 2) 


# Management is not involved 


> Fails to provide leadership 
¢» Needs to be actively involved 


«> Needs to “crack-the-whip” 

@ Organizations not ready for PI (not addressed herein) 

> If organizations are not ready for change PI will fail 

> All units in an organization need to want to improve 

@ Supplier mature in process but not acquirer (not addressed herein) 


> Itis very difficult for a contractor to be Level 2 or Level 3 if the 
acquisition Agency is Level nothing 
> A project will perform to the lowest level of either party 
«> The lower level party brings the higher level one down to its level 
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Real Examples of Impediments (1 of 4) 


Example 1 


@ An organization attempting Level 3 

> Management schedule is 6 months 

> Organization has no documented processes 

> Organization is not following many processes 

> Management fails to provide enforcement or support 

> Organization fails to achieve even Level 2 in six months 
> I quit 


Example 2 


@ Ata CMM conference 

> Overheard at dinner table about a company involved in PI 

> Someone asked how far they had gotten 

> In one year they had implemented a process for conducting meetings! 
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Real Examples of Impediments (2 of 4) 


Example 3 


@ Large acquisition agency 
> Implements Quality Assurance (QA) 
> Lacks QA expertise 
¢» Extensive training and mentoring required 
3 MITRE staff - 80%, 3 Government staff - 300% 
Insists on extensive process documentation 
«> About 7 procedures for QA each about 13 pages 
¢ Extensive guidelines and training 
*« Write, review, re-write, re-review... (change “happy” to “glad”) 
* Preciosity sets in 


V Vv 


The MITRE Corporation 7 


Real Examples of Impediments (3 of 4) 


Example 3 (comp.) 


> Reorganizes several times 

*«» Each time new managers and staff need to agree on QA approaches 
«» Each time new managers and staff need to be trained 

> After about 9 months QA process in place 

> Organization reorganizes several more times 

“> New mangers and staff need to accept QA and be retrained 

> QA falls apart 

> QA effort restarted again two years later from scratch 


>» Exactly the same scenario is experienced in attempting 
to implement Configuration Managment (CM) 
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Real Examples of Impediments (4 of 4) 


Example 4 


> Project plans 

System Engineering Management Plan 
Software Development Plan 
Configuration Management Plan 
Quality Assurance Plan 

¢ Etc. 

Often takes months to develop 

Involve tons of people 

Cost in excess of $100K! 


| have experienced this many times on different projects at different 
corporations 
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FASTrack to Process Improvement (1 of 5) 


@ A solution to PI impediments may be in FAS Track 


> A process that is in need of time critical improvement is quickly 
implemented without much of the fanfare and tedious 
documentation that bogs down PI efforts 

> The process is quickly defined and executed immediately as a pilot 

> During the pilot's execution, the process is refined, as necessary, 
and the process documentation is enhanced as required 

> Data and artifacts are collected on the pilot’s execution which may 
be used as proof that the process is, or is not, successful and for 
continued process improvement 

> Additional FASTrack pilots are executed as necessary throughout 
the PI efforts 
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FASTrack to Process Improvement (2 of 5) 


@ FASTrack implementation provides: 

> Early results to management 

> Buy-in from senior management and staff 

> Momentum to continue 

> Early process execution and artifact collection 
> Proof that process works 

> On-the-fly maturing of the processes 


> PI done for business sake 
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FASTrack to Process Improvement (3 of 5) 


@ Identify project/process for FAS Track 
® Ifa process exists 
> Tailor and use 
@ If process does not exist quickly: 
> Develop draft 
> Process procedure 
> Procedure templates 
> Procedure guidelines 
> Procedure training 
> Select Standards 
@ Execute process 
> Start when draft procedure is developed 
> Continue as other drafts are done 
Collect artifacts 
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FASTrack to Process Improvement (4 of 5) 


Mature process (on-the-fly) 
Expand 
> Procedure 
> Templates/guidelines 
> Training 
#@ Develop/Expand Process Policy 
Select other Standards 
Store in Process Asset Library (PAL) 


@ 27, 37, and subsequent FASTracks can begin anytime after the 
initial ones have started 
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FASTrack to Process Improvement (5 of 5) 


—— a 
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Real Examples of FASTrack (1 of 19) 


Example 1 


Contractor developing the Galileo Probe to Jupiter 

> Micro processors, assembly language, firmware based 

@ Had never developed space firmware 

Had estimated effort at 3 developers for 6 months 

> Thought it was equivalent to a small FORTRAN program 
@ Had never developed real-time assembly/firmware code 
@ Had no software processes or standards 
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Real Examples of FASTrack (2 of 19) 


Example 1 (cont) 


@ As the software development manager, | quickly 

> Developed (in about 3 weeks) a software process compliant with 

many CMMI Level 2 processes 
«» Documented in a development plan 

> Re-estimated the effort 

> Acquired qualified staff 
# Management was shocked that software could follow an 

engineering process 


@ Galileo spacecraft and probe were very successful for NASA 
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Real Examples of FASTrack (3 of 19) 


Exam p I e 1 (comp.) 


“Al has generated software design, methodologies, design criteria and 
configuration control procedures for Galileo that will be a model for all 
programs. These procedures are unique in their completeness and 
quality for the Corporation for small software projects.” G. A Kravetz 


“Many of us in the program office were originally concerned that the 
newness of flight software and the specific complexity of the software 
would compromise our estimated cost and schedule. The well organized 
software approach that Al has formulated and implemented has 
completely removed any concern. It should be noted that the software 
management plan generated by Al has been adapted for the entire Galileo 
program including the flight software for the relay receiver and the system 
test software.” R. J. Drean 
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Real Examples of FASTrack (4 of 19) 
Example 2 


@ Management for a DoD contractor would accept new 
requirements from the acquisition agency (Theatre Battle 


Management) without conducting impact assessments and 
would instruct staff to implement them 


Staff was working overtime and on weekends to keep up with 
this requirements creep 


| was new to the organization as a senior systems engineer 
| convinced management that they needed to evaluate the state 
of their engineering/management practices 
> They seemed unaware that things could be done different 
@ They allowed me to attempt to fix this problem 


@ | quickly developed a CM process, in two days, which included a 
Configuration Control Board (CCB) 


@ | quickly trained staff and management on the process 
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Real Examples of FASTrack (5 of 19) 
Example 2 (cont) 


@ The next requirements request from the customer was given to 
me to evaluate 


@ | had staff: 
> System > Quality Assurance 
> Hardware > Configuration Management 
> Software > Program Management 
> Test > Contracts 
Conduct an impact assessment against their interests not 
limited to: 
— Function 
— Performance 
— Cost 
— Schedule 


Interfaces 
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Real Examples of FASTrack (6 of 19) 
Example 2 (comp, 
The impact assessment showed that the request could not be 
implemented 
| told the CCB that we had to reject it 
> Can we do that? 
> Watch me! 
@ CCB recommended rejection 
# Management accepted the recommendation 
> This was the first time they had ever rejected a customer request 


@ This started the organization down the path to process 
Improvement 
> Within a few years they had achieved CMM® Level 3 
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Real Examples of FASTrack (7 of 19) 
Example 3 


Contractor developing a very large criminal database application 
@ Going into a System Design Review 
@ As the newly assigned Development Manager | reviewed the 
System Requirements Specification 
> Requirements very poorly written: ambiguous, inconsistent, not 
uniquely identified, untestable, unimplementable, etc. (So what else 
is new?) 
@ Convinced management and the customer to delay the review 
by one month to fix specification 
@ Quickly documented a methodology for the proper specification 
of requirements?+ 
> Had been using this method since 1978 
Trained the system, software, and test engineers on the 
methodology 
We rewrote the specification and passed the review without any 
action items assigned to us The MITRE Corporation 23 


Real Examples of FASTrack (8 of 19) 


Example 4 


# Same contractor developing a very large criminal database 
application 

Configuration Control Board (CCB) meeting once a week for 2 to 
3 hours 


> CCB items ranged from requirements to design defects and every 
thing in between (even the kitchen sink) 


> Membership included all mangers and all technical leads 
¢* About 12 members 
¢» Program Manager (PM) Chair 
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Real Examples of FASTrack (9 of 19) 


Example 4 (ont) 


@ | convinced the PM to allow me to redesign the CCB and 
Configuration Management Process 
> CCB items ranged from baselined requirements to other important 
contractual, critical issues such as: 

«» Statement-of-Work 
* Master Schedule 
«» Costs, based on thresholds 
> CCB met monthly or as necessary 
> CCB membership 

¢* PM (Chair) 

¢«* Deputy PM 

¢« Development Manager 

« Systems Engineering Manager 

¢ Test Engineering Manager 

¢«* CM and QA Managers 

“» Contracts (as necessary) he Mase Corporation: «25 


> 


> 


o 


> 


> 


Real Examples of FASTrack (10 of 19) 
Example 4 (comp.) 


> Formed a Technical Review Board (TRB) 
“> Winnow out items from CCB 

«» Promote appropriate items to CCB 

«» Conduct impact assessments on baseline changes 
¢» Make recommendations to CCB 

> TRB met weekly 

> TRB membership 

¢» Deputy PM (chair) 

¢« Development Manager 

¢» System Engineering Manager 

¢ Test Manager 

“«* CM and QA Managers 

«» Technical Leads (as required) 


> 


G 


S) 


> 


G 
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Real Examples of FASTrack (11 of 19) 
Example 5 


@ A DoD organization that acquires and also develops products is 
implementing both the CMMI® for development and the CMMI® 
Acquisition Module for acquisition 


@ Using a FASTrack approach 


Formed 

> Steering Committee (Senior Executive Management) 
«» Approve, release, and enforce process policy 
«> Approve process procedures 
¢» Oversee and approve Process Group’s efforts 

> Process Group (3 MITRE staff 60% FTE, 5 DoD staff 25% FTE) 
« Provide status and recommendations to Steering Committee 
«> Develop/acquire processes 
«» Develop/acquire and present process training 
« Select projects for FASTrack 
«» Select processes for FASTrack 
¢» Follow FASTrack approach 
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Real Examples of FASTrack (12 of 19) 


Example 5 (cont) 


@ Process Group 
> Developed Process Policy 
> Selected pilot project 
> Selected processes to pilots 
> Developed Schedule 
«» Level 2 Appraisal, 
«» Level 2 and Level 3 processes 
¢» Acquisition processes 
> Developing draft processes 
> Developing draft guidelines 
> Developing training on processes 
> Providing training 
Steering Committee 
> Signed Process Policy 
> Reviewed and approved FASTrack approach 
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Real Examples of FASTrack (13 of 19) 


Example 5 (cont) 


@ Pilot Project 

> Taking training 

> Applying FASTrack on processes 

> Maturing processes 
Organization had no documented procedures 
@ Pilot project followed undocumented methods and had some 

process artifacts 

> CCB Minutes 

> CM Plan 

> Baselined Requirements 

> Etc. 
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Real Examples of FASTrack (14 of 19) 


Example 5 (cont) 


Leveraged *CMMI® to develop draft procedures and training 
> Two or more processes developed and implemented per month 
(note: 9 months to implement QA shown on Examples of 
Impediments, example 3, with many more resources) 
¢«» Developed draft procedure and guidelines 


«» One to two procedure per process area (3 to 4 pages vs. several 
procedures with up to 13 pages shown on Examples of Impediments, 
example 3) 


«» Developed and delivered process training (can be uses as guidelines) 
@ Piloting procedures on project 

> Provide mentoring 

> Mature process 

> Collect artifacts 


*Credit given to SEI 
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Example 5 cont) 
1. 


Introduction 

Purpose 

Scope 

Plan Organization 

Related Documents 

Tailoring this Plan 

Plan Maintenance 

Quality Goals and Objectives 
Quality Assurance Program 
Process Steering Committee 
Process Group 
Enterprise-Level Quality Assurance 
QA Manager 

QA Personnel 
Program/Project Management 
Programs/Projects Personnel 
QA Measurement and Metrics 
QA Risks 

QA Resources 

Facilities and Infrastructure 
Tools 

Costs 

QA Training 

Senior Management Orientation 
Program/Project Staff 

QA Staff 
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QA Plan Tailoring (1 oF 4) 


APPENDIX A QUALITY ASSURANCE TASKS - TAILORING TABLE 
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Real Examples of FASTrack (16 of 19) 
Example 5 cont) QA Plan Tailoring (2 of 4) 


Appendix A 


Describes the activities to be executed to effectively administer 
the QA program. It provides a tailoring schema used to tailor 
this plan to be used on specific programs and projects. 


Allows for quickly adapting plans to projects 
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Real Examples of FASTrack (17 of 19) 


Example 5 cont) QA Plan Tailoring (3 of 4) 


Appendix to QA Plan 
QUALITY ASSURANCE TASKS 
TAILORING TABLE 
QA tasks are shown in the table along with task numbers. Responsible organizations are 
shown along with QA process areas. References to procedures are included for those tasks 
that require them. The table allows programs and projects to tailor the QA process to the 
scope of a specific application by indicating “Y” for “used” and “N” for “not used”. Additional 
QA tasks can be added at the end of each section, if needed, for a specific application. The 
main body of this plan should not be tailored since all QA tasks are included in this appendix. 


TAILORED FOR PROJECT/PROGRAM: 
Program/Project Manager: 

Date: 

OTC Process Group: 

Date: 

OTC Enterprise-Level Quality Assurance: 
Date: 

OTC Process Steering Committee 

Date: 
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Real Examples of FASTrack (18 of 19) 


Example 5 (con) Appendix QA Plan Tailoring jor “NN 


3.4 QA Manager 


Primarily responsible for overseeing QA task assigned to programs/projects and to QA personnel + 
b. | Works closely with the Process Group and serves on the group whiled developing QA process artifacts and 
QA orientation and training 


Cc. Provides QA orientation and training — 
d. Works with Program/Project Management in tailoring and applying QA. a 
e. Attempts to resolve QA non-compliance with program/project management _ 


f. Escalates QA non-compliance to Steering Committee = 
3.5 QA Personnel i 
Support the development/acquisition of QA processes and best practices — 


Support |b. Support the development and tailoring QA plans |b. Support the development and tailoring QA plans and tailoring QA plans 


Conduct product reviews on external products delivered by contractors and suppliers 
QA Product Review Procedure 


Conduct sis aca reviews at the enterprise level and on programs/projects 
sis aca Product Review Procedure 


e. Conduct process audits at the enterprise level and on programs/projects 
1. QA Process Audit Procedure 
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Real Examples of FASTrack 19 of 19) 
Example 5 (comp) Level 2 Appraisal in February 07 Scheduled Sek ee ae 


Category Process Area [Level] CMMI® {CMMI®-AM | Timeline 


Project Planning [2] 


Project Project Monitoring and Control [2] hed 7 
Management | Supplier Agreement Management [2] Sep 06 
Integrated Project Management (IPPD) [3] Oct 06 
Integrated Supplier Management (SS) [3] Oct 06 
Integrated Teaming (IPPD) [3] Dec 06 
Risk Management [3] July 06 
Quantitative Project Management [4] N/A 
Solicitation and Contract Monitoring [AM] Sep 06 
Configuration Management [2] Feb 06 
Process and Product Quality Assurance [2] Mar 06 
Measurement and Analysis [2] June 06 
Decision Analysis and Resolution [3] Jan 07 
Organizational Environment for Integration (IPPD) [3] 
Causal Analysis and Resolution [5] N/A 
: : Requirements Management [2] Jan 06 
Engineering Requirements Development [3] Jan 06 
> Technical Solution [3] Aug 06 
Product Integration [3] Jan 07 
Verification [3] Aug 06 
Validation [3] Aug 06 
Transition to Operations and Support [AM] Dec 07 
Process Organizational Process Focus [3] YES Feb 07 
Management | Organizational Process Definition [3] YES Feb 07 
Organizational Training [3] YES July 06 
Organizational Process Performance [4] YES N/A 
Organizational Innovation and Deployment [5] YES NO N/A 
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Summary (1 of 2) 


@ CMMI implementation can be: 
> Costly 
> Time consuming 
> Prone to failure 
@ Many reason for this 
Management support 
Schedule constraints 
Funding 
Immediate expectations 
Excessive process documentation 
Failure to get buy-in 
Etc. 


VV VV VV WV 
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Summary (2 of 2) 


@ FASTrack provides 


> 


VV VV WV 


Early results to management 

Buy-in from senior management and staff 
Momentum to continue 

Early process execution and artifact collection 
Proof that process works 

On-the-fly maturing of the processes 
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Introducing the RUP Platform 
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Basic Structure of RUP: Key Concepts 
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Tool 
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Checkpoints Template Report 
Refers _ 
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Roles Perform Activities and Produce Artifacts 


Business yjsion Stakeholder Vision _ Requirements 
Rules Requests (refined) app emenialy Management 
— < e a Plan 
ARES oo | 
a i Requirements 
ributes 
Example: © 
Requirements-> yy, L> -, 
: Vision Dependencies ) 
Workflow Detail-> Sisiir Requirements 


- Analyst Attribut 
Define the System nee | > | > (refined). 
Capture a Find Actors I Ss 


Common and Use Cases 
Use-Case Model 


Vocabulary Wa 
a = (refined) 
= ‘Te tt GD 
Use-Case Q— 

Modeling Business Use Case 
Glossary Guidelines Object Model Use-Case Model (Outlined) 


(refined Business 
Use-Case Model 


Ill 


Glossary 
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The Spirit of RUP 


1. 


2a oe S&S PL 


2002-20 


Attack major risks early and continuously... or they 
attack you 


Ensure that you deliver value to your customer 
Have a maniacal focus on working software 
Accommodate change early in the project 
Baseline an executable architecture early on 
Build your system with components 

Work closely together as one team 

Make quality a way of life, not an afterthought 
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Iterative Development 


Disciplines 

Business Modeling 

Requirements 

Analysis & Design 

Implementation 

Test 

Deployment 
Configuration 

& Change Mgmt 


Project Management 
Environment 


Tran || Tran 
#1 #2 | 


Tarations 
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What Will RUP Get You? 


¢ A collection of world best practices 


¢ Amechanism for modifying them to suite your needs, as 
well as integrating other best practices and your own 
practices 


¢ A mechanism for tailoring and deploying them to 
projects 

¢ With Rational Portfolio Manager, a mechanism for 
managing projects according to projects’ defined 
processes, and gathering important data critical to data- 
driven project management & future planning 
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What Will RUP Not Get You? 


¢ A guarantee that you'll perform better without some 
effort on your part 


¢ Instant expertise in software engineering best practices 


¢ Compliance with all CMMI maturity level 2 & 3 
requirements out-of-the-box 
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How Does RUP Fit In? 


¢ The key to realizing benefits is 


— implementing organizational standard software engineering 
processes 


— that are deployed (franchised) into projects based on key 
characteristics 


— and are tailored, managed & measured 
¢ A great way to start is a combination of 
— Rational Unified Process (RUP) 
— Rational Method Composer (RMC) 
— Rational Portfolio Manager (RPM) 
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¢ Use of the Rational Unified Process and IBM Rational 
automation tools will accelerate the achievement of 
CMMI maturity level 3 


| Tailor / Improve | Institutionalize | Achieve CMMI | 
Processes | Processes maturity level 3 
“5 13 _h. i 


aA 
7 Z - 
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Core RUP RMC RMC 


Phases _ {ww on acan N | Large 
Disciplines | | tnception| Elaboration Construction Transition eo} — — | J2EE 
= r | ee re | Project 
= ee, ars Sas | 
iH i ey a 
SS T Te a Small 
H an neatouh | Team 
coe | a ee (se a Project 
Inti Elab #1 Blab #2 Cone re sy Ter 7 Y 
Iterations [2.3 


Development Process engineers, : 
‘or : Project managers 6 
organization elcelele-lanls e)ae)(sreqs Practitioners 
: & team leads 
offices 
Common Process Process Process 
methodology authoring configuration delivery 
>» Shared understanding >» Leverage internal >» Configure and deploy >» Manage project 
of terminology, knowledge and process for specific instances 
deliverables, and process assets tools, technologies, > Gather and report 
responsibilities domains attics 
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RUP & CMMI 
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Situation Description 


¢ Evaluated RUP for Large Projects v7.0.1 against CMMI- 
DEV w/IPPD v1.2 maturity level 3 


¢ RUP’s processes, tasks, templates, etc. were reviewed 
and evaluated for compliance 


— Determined the risk of meeting the CMMI’s intent in each ML 
2 &ML 3 practice if the process & associated templates are 
implemented as described 


— Use of Rational tools was not considered 


¢ Caveat: Actual results might vary based on project 
implementation and degree of RUP tailoring! 
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Project Management Process Areas 


Strengths 
— RUP fundamentally provides a solid project management 
foundation 
¢ Planning, Tracking & Oversight, Risk Management 
Areas for Improvement 


— Supplier Agreement Management — RUP does not address 
SAM or the notion of suppliers at all 


— Estimation and tracking of estimates leaves too much 
discretion to the project 


— Planning for and managing project data — too code-centric 
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Engineering Process Areas 


Strengths 
— Overall, RUP’s Requirements, Design & Analysis, 
Implementation, Test, and Deployment disciplines handle 
CMMI requirements very well 
Areas for Improvement 
— Traceability without tool support may not be sufficient 
— Determining criteria for alternative technical solutions 
— Performing build/buy/reuse analysis 
— Integration sequence, procedures, and criteria 
— Reviews of interfaces for coverage & completeness 
— Analysis of peer review data 
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Support Process Areas 


Strengths 
— RUP’s Configuration & Change Management discipline 
— Measurement & Analysis 


Areas for Improvement 


— Decision Analysis & Resolution - RUP does not provide a 
process for DAR 

— Process Quality Assurance Is not called out strongly enough 
— projects may overlook this important process area 

— Identification of configuration items Is too focused on 
engineering work products and doesn’t address other work 
products 
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Process Management Process Areas 


Strengths 


— RUP’s collection of best practices and mechanism (Rational 
Method Composer) for tailoring and deploying to a project 


Areas for Improvement 


— All Process Management guidance is project-centric - RUP 
does not address organizational-level processes for 
¢ Organizational Process Focus 
¢ Organizational Process Definition 
¢ Organizational Training 
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Generic Practices 


Strengths 


— RUP’s collection of best practices and mechanism (Rational 
Method Composer) for tailoring and deploying to a project 
Areas for Improvement 


— RUP does not provide any policies or guidance for policy 
development 


— Identification, planning, scheduling, and involvement of 
Stakeholders 


— ldentification and delivery of personnel training 

— Feedback loop of project-level measurements and lessons 
learned to improve the organization's processes and 
planning capabilities 
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TOLL ALefmerelele Mm eLUl Ma (0) Mm DlOM I (-Me1-) am Mal-)a-Wararg 


Implementation of software engineering best practices and 
integrated automation tools across an organization Is 
difficult 


Barriers 
— Lack of understanding 
— Poor planning 


— Resistance to change 


“Radical, rapid change Is crucial to success; incremental, 
unfocused changes have a better than 50% chance of failure” 


— Gartner 
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RUP Implementation Roadmap 
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Roadmap - Setting the Stage 


1. 


Establish Executive Sponsorship with the expectation 
it is active, not passive 

Clearly tie the effort to business goals 

Establish a guiding coalition (MSG/EPG/SEPA) of 
movers and shakers from across the organization to 
drive the RUP implementation strategy, approach, and 
plan 

Projectize the effort, assign a cost center, and treat it 
like a project with clear milestones and reviews 


Tie improvement objectives to each individual’s 
performance review 
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Roadmap - Tailoring RUP - 1 


1. Using Rational Method Composer, develop 
organizational processes for: 
DAR, OPD, OPF, OT, SAM 


2. Create detailed tasks for: 


11/16/2006 
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Policy development 

Estimation & tracking of actuals 

Identification and delivery of project-level training 
Objective evaluation of processes 


Feedback of measurement and lessons learned to organizational 
process assets 


Build/Buy/Reuse decision making 
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Roadmap - Tailoring RUP - 2 


3. Improve existing guidance for: 
¢ Management of non-engineering Cls/project data 
¢ Stakeholder involvement 
¢ Traceability 
¢  Alterative technical solution criteria & evaluation (DAR) 
¢ Evaluation of peer review data 
¢ Integration sequence, procedures, & criteria 
¢ Interface reviews 


4, Using critical thinking, determine the various RUP 
configurations your organization might use and 
establish them as approved lifecycles in RMC 
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Roadmap - Introducing Improvements 


1. Determine gaps in the knowledge and skills required 
in the organization, train people 


2. Pilot on carefully selected projects, provide 
hypersupport 


3. Identify pilot project lessons learned and incorporate 
improvements into the organization’s processes 


Iteratively deploy more broadly 
5. Maintain the feedback loop, continuously improve 
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¢ The outcome will be an integrated, organizationally 
cooperative process infrastructure (OSSP) that: 


— is the foundation for a successful organizational 
transformation and improvement 


— provides common processes for measurements & analysis 

— adopts industry best practices as embodied in RUP 

— facilitates software improvement based on consensus 
priorities 

— Delivers superior engineering results 
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Questions/Discussion 
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Major Milestones: Business Decision Points 


Customer 
acceptance 
or end of life 


Product sufficiently 
mature for customers 
Commit resources 
for construction 


Commit resources 
for the elaboration 


phase N\ 


Tater} ola (oyal =i relete) e-tateya| Construction Transition 


iit A A A A 
Lifecycle Lifecycle Initial Operational Product 
Objective Architecture Capability Release 
Milestone Milestone Milestone 
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2. Ensure That You Deliver Value to Your Customer 


¢ Focus on key requirements 
— Capture, document 
— Organize, prioritize 

¢ Requirements will change 


— Evaluate impact of change and decide what changes 
to implement 


— Propagate changes to all team members 
¢ Make requirements accessible 


Requirements management leverages 
your ability to deliver products 
that meet user needs 
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« Ause case describes complete and meaningful services that 
your system offers to users and other systems 


¢ Use cases drive the work through each iteration 
— Planning of iterations 
— Creation and validation of the architecture 
— Definition of test cases and procedures 
— Design of user interfaces and creation of user documentation 


ae, 
Use-Case Model 
: : a by 
Analysis & Design Model =e 
: implemented by 
) Implementation Model = =] 


Test Model 


verified by 
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3. Have a Maniacal Focus on Working Software 


¢ Measure progress primarily by reviewing executable 
code, and test results 
— Plans, requirements, designs and other by-products often 
provide a false perception of progress and status 
¢ Focus on the final, delivered product, and only the 
artifacts that matter to get at this goal consistently 
— Streamline the process 


— Do not use all of the RUP! Only use what makes sense to 
your project — tailor it to suite your organization’s and 
project's needs. 
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¢ Today’s systems are too complex to get the 
requirements, architecture, design, implementation and 
scope right the first time 


Accommodate changes in the: 


business architecture design & implementation scope (reduction) 
solution 


eee] ain | cnn 


Preliminary Architect. Architect. | Devel. Devel. Devel. Transition Transition 
Iteration Iteration Iteration Iteration Iteration Iteration Iteration __ Iteration 
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¢ Architecture provides a skeleton structure of your system 
— Subsystems, key components, interfaces, architectural mechanisms 
(solutions for common problems, such as persistency, inter-process 
communication, ..) 
¢ Implementing and testing the architecture mitigates most 
technical risks 


Produce executable architecture 


ae 


Preliminary Architect. Architect. | Devel. Devel. Devel. Transition Transition 
Iteration Iteration Iteration Iteration Iteration Iteration Iteration __ Iteration 
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¢ Architecture-driven, Component-based Design 
¢ Component architecture provides flexibility 


System Software Layered, Component-based 
Requirements EB. Architecture 
-B 
=a 
R1 
R2 Rj pA 
RN Rk | coniponents 
Rx Common 
Ry | Mechanisms 
Rz 
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7. Work Closely Together As One Team 


¢ Empowered and self-managed 

— Clear vision 
« Accountable for team results 

— Clear expectations 

— All for one, one for all 

- avoid “My design was good, your code didn’t work” 

¢ Optimized communication 

— Effective process (right-sized for your project) 

— Organize around architecture, not around functions 


— Get the right tool support 
¢ Easy access to current requirement 
¢ Private workspaces 
¢ Easy access to defects... 
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8. Make Quality a Way of Life, Not an Afterthought 


Software problems are 
100 to 1000 times more costly 
to find and repair after deployment 


¢ Cost ito Repair Software 
¢ Cost of Lost Opportunities 


Cost ¢ Cost lof Lost Customers 


ee 


Inception Elaboration Construction Transition 
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Customer Success Is Our Mission 


Right Turns and Wrong Turns to 
Level 5 


Thomas Lienhard 
Raytheon Missile Systems 


Timothy Davis 
Retired — Raytheon Missile Systems 


Special Thanks — Kathy King 
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Agenda 


> Typical views of organizational and project roles for High 
Levels of CMMI Maturity 
> Risks in using these roles in certain environments 


— Using homogenous assumption in non-homogenous 
environments 


— Assumption that each project contributes equally to 
organizational goals 


> Improved organizational and project statistical 
management approach 


> Actual example 
> Translation into QPM and OPP terms 
> Summary 


Raytheon 
Typical View of Organizationader ss: oumsio 
Role at High Maturity 


> Organizational quantitative business goals and 
objectives are injected as project objectives 

> Focus on development part of lifecycle 

> Organization’s process performance baselines 
and predictive models are established to be 
common and applied to all projects 

> Process performance baselines consist 
of highly aggregated data 


: : : Raytheon 
Typical View of Project Role «me:siccess'outision 
at High Maturity 


> Project objectives are “given” by the organization 

> Selection of critical processes and product parameters 
— Directed by organization, or 
— Performed once, during planning phase 

> Projects are managed using organizational process 
performance baselines and models 

> Projects contribute project measurements of sub- 
process measurements to refine existing 
organizational process performance 
baselines and models 

> Special cause analyses assume models 
and baselines are applicable to all projects 


O rg an iZall O nal Chan g es D UV Qustomer success ls Our Mission 
Implementation Changes 


> Organizations are becoming less homogenous 
— Expanding boundaries 
e Multiple sites 


a Changing 
e Strategic acquisitions Traffic 
e Multiple disciplines Pattern 
— Breadth of organizational responsibilities Ahead 


growing 
¢ Development 
e Sustainment and Maintenance 
e Services 
> Product development is evolving 
— New products are based on existing products 
— Cost and profitability are critical components 


Interpretation of High Maturity needs to be revisited as well 
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Today’s Environment 


> The business model is now to provide complete 
solutions not just components, disciplines, etc. 


> Development is only small part of solution (lifecycle) for 
more and more programs 


> Models generally need iterative refinement to incorporate 
real data as it becomes available as the product matures 


> Project solution space extends beyond technical 
requirements, to include business factors like cost, 
affordability and producibility 


Dev Manufacturing/Assembly 


Product Shelf life 
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Product Costs Must be Addressed Early 


i * Design Influences 
a . 70 Percent of Initial 


ae ? Product Cost 


% 
INFLUENCE 
ON TOTAL 


Courtesy of 
Munro and Associates 
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Potholes With Typical Approach - 1 


> Organizations tend to search for commonalities that transcend each 
project 
— Common lifecycle phases may not share same criticality 


— Models tend to be single dimensional — cost, affordability and 
producibility not integrated with technical performance parameters 


— Aggregated data from multiple projects increases variability thus 
reducing usefulness 


— Unique objectives for projects not always taken into account 


> Organizational models and baselines fail to provide needed project 
insight 
— Expected range typically not tight enough to aid projects 
— Subprocesses which are statistically managed 
aren't the ones that are key to project success 
— Lack of rational subgrouping leads to 
baselines common to all but useful to none 
— Models are not always calibrated/rerun 
as actual data becomes available 
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Potholes With Typical Approach - 2 


> Not working for the project 
— Same allocation of organization objectives to all projects 


— Having “directed” subprocess stable and capable doesn’t always 
guarantee success 


— Not enough insight into what is key 
— Takes time and money to chase non-critical measures 


— Focus is diverted away from what Is truly critical to project 
SUCCeSS 


> Not working for the organization 
— Customers aren't always satisfied 


— Not enough insight into what needs 
to be fixed 


Project and organizational achievement 
of business goals is sub-optimized 
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Understanding What’s Critical 
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The Idea is to Predict Performance Early...... 


> 


> 


Understand the voice of the customer — what is this product’s 
objectives? 
Determine product characteristics that affect the voice of the customer 
— Key Product Characteristics (KPC) 
Understand what influences the KPCs 
— Key Control Characteristics (KCC) 
Use existing methods, models and simulations to determine where to set 


the KCCs to maximize probability of achievement of the product’s 
objectives 

By statistically managing these, it will be possible 

to predict whether the project will be able to achieve 

its objectives 


....O increase the probability of meeting project 
requirements, at reduced cost and in less time. 
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Identify Key Control Characteristics 


Programs identify Key 
Product Characteristics 
(KPC) 


Continued 
identification of 
KPCs to lower 
levels through 
Functional 
Analysis 


Finally identify 
Key (Process) 
Control 
Characteristics 


(KCC) Simple Example of a KCC Affecting Several KPCs 
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Example of Key Characteristic Development 
System Level KPC 


| Probability of Launch | Probability of Probability of Hit 
Acquisition 
| Other Subsystem KPCs | Guidance Section Sensitivity Other Subsystem KPCs 
Transmitter Power Antenna Pattern 


KPCGs Null Depth 


Mask Trace Routing Bondline Plating 
Alignment Dimension Dimensions Thickness Thickness 
Etch Time Etchant Bath Bond Pressure Film Viscosity 
KCCs Concentration Temperature 


System Level KPC driven by lower KPCs and KCCs 


R 
TURN 
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Quantitative Project Management 


IPT input, Org baselines 
specifications | | | | and models 
Customer provides 
product 
performance —— 
requirements 
Project Design: 
>| Modeling > Create prototypes 


Requirements (KPCs, KCCs) 


Organization A 
allocates business }—— 
requirements 
Project 
4 [ pee neaee [ 
\ models \ 
ade A Vv 
> Statistical methods are used to 
identify critical subprocesses and ys Meets required 
. . characterization? 
component objectives for each 
project uf 
. . . bd 
> CAR isa integral part of planning 
and management | Implement product 


> Baselines may be project unique 


Helps From Design through er sicessisoumsin 
Manufacturing, All the Way to End User 


> Helps managing the design complexity of a large system, leading to a robust design, by 
concentrating on the features which significantly impact key objectives 


> Forces attention at those design aspects that are most critical to satisfy Customer 
needs, 


> Ties Critical Parameters on lower design levels back to customer needs and quantify 
their impact, 


> Improves design quality by driving a “whole system view” mindset throughout the 
design teams and increases predictability/reduces risk for integration, 
design/development and manufacturing. 


> Increase predictability/reduce risk for design/development, integration, mfg and mission 
SUCCeSS. 


> Reduces overall development cycle time and cost 
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Summary - 1 


> Organizations with homogenous projects benefit 
from organizational commonality for quantitative 
management 
— Selecting critical subprocesses may 
be pre-determined 
— Baselines and models already exist 
and only need calibration with use 
— Lessons learned and improvements 
translate easily to projects 
throughout the organization 
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Summary - 2 


> Organizations with non-homogenous projects 


— Organization objectives need to be allocated to individual 
projects/programs 


— Projects identify component characteristics and subprocesses whose 
variation has greatest impact on all project/program objectives 


— As real data is available models are recalibrated 

— The entire product life-cycle needs to be taken into consideration 
> Organizational role 

— Provide baselines and models at component level 


— Manage Organizational Set of Standard 
Processes (OSSP) to meet organizational 
objectives 
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Contact: 


Thomas G Lienhard@Raytheon.com 
Timothy_J_ Davis@WillWork4U.com 


Top Software Engineering Issues 
in the Defense Industry 


NDIA Systems Engineering Division 
and Software Committee 


September 26, 2006 


NDIA Systems Engineering Division - Top SW Issues i 
August 24-25, 2006 


Task Description 


Identify Top 5 Software Engineering problems or issues 
prevalent within the defense industry 


¢ NDIA Workshop August 24-25, 2006, Washington D.C. 


Document issues 

¢ Description and current state 

e Rationale and SW impacts 

¢e Develop recommendations (short term and long term) 


Generate task report 
¢ Submit to OSD 
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STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Task Group Participants aah 


26 workshop participants from industry, government, academia 
¢ Additional inputs received via email 


BAE Systems Massachusetts Institute of Technology (MIT) 
The Boeing Company The MITRE Corporation 

Center for Strategic and Intl. Studies Naval Air Systems Command 

Computer Sciences Corporation OUSD (A&T) 

Department of Defense (DoD) Open Systems Joint Task Force 

Defense Contract Mgmt. Agency (DCMA) Raytheon Company 

Florida Institute of Technology RDECOM ARDEC 

Harris Corporation Asst. Secy. of the Air Force (Acq), SE Division 
Jacobs Technology Software Engineering Institute (SEI) 
Lockheed Martin Corporation Systems and Software Consortium 


U.S. Air Force Aeronautical Systems Center 


Task Group Leads: 
Geoff Draper (Harris) 
Jeff Dutton (Jacobs Technology) 
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‘STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Task Group Activities swap mah 


Top 5 SE 
Issues 
(2006) Candidate Top 5 SW Task 
issues Issues report 
Status of 
OSD 
Initiatives .Background *30 consolidated root issues  *Extended to 7 key issues 
Brainstorming Grouped and refined *Detailed description 
*Group discussion eEffects and root causes «Recommendations 
*85 root issues identified *Consensus review, voting 
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August 24-25, 2006 : 
Version #1.0 


Common Themes Noted 


Increasing demands for rapid response to changing threats 
¢ New warfighter capabilities 
e Unprecedented complexity 
e Increasing dependence on large, software-intensive systems 


Evolving acquisition environment 

e Innovative and aggressive acquisition strategies 

e Extensive reuse of legacy components and system portfolios 

¢e Reductions in acquisition work force due to force-shaping 
measures 


Current approaches for acquiring, developing, verifying, 
and sustaining software enabled systems are inadequate 
to deal with the complexities of a dynamic and changing 
acquisition environment. 
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Top Software Issues 


1. The impact of requirements upon software is not consistently 
quantified and managed in development or sustainment. 


2. Fundamental system engineering decisions are made without full 
participation of software engineering. 


3. Software life-cycle planning and management by acquirers and 
suppliers is ineffective. 


4. The quantity and quality of domain-knowledgeable software 
engineering expertise is insufficient to meet the demands of 
government and the defense industry. 


5. Traditional software verification techniques are costly and 
ineffective for dealing with the scale and complexity of modern 
systems. 


6. There is a failure to assure correct, predictable, safe, secure 
execution of complex software in distributed environments. 


7. Inadequate attention is given to total lifecycle issues for 
COTS/NDI impacts on lifecycle cost and risk. 
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Issue 1: The impact of system requirements upon aT 
software is not consistently quantified and managed War nEe 
in develo ment or sustainment. STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Discussion Points: 
e Impacts of changes to requirements not addressed consistently. 
¢ SW requirements are not always adequately addressed in system level 
requirements or solicitations/contracts. 
¢ Conditions leading to SW reqts changes are not clearly understood. 
¢ Weak linkage to SW requirements for capabilities, portfolios. 


Recommendations: 
Enforce effective software requirements development and 
management practices, including assessment of change impacts, 
for both the acquirer and supplier organizations. 
e Strengthen policies/guidance for full requirements traceability. 
e Evaluate effectiveness of models for estimating SW and changes. 
¢ Improve involvement of SW and acquisition expertise 
- Early requirements definition (JCIDS) 
- Balance user requirements with cost/schedule constraints 
- Impact assessment and communication of requirement changes 
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Issue 2: Fundamental system engineering decisions 
are made without full participation of software 
engineering. 


Discussion Points: 

e SW not consistently involved in architectural decisions early in the life cycle. 

e Must have authority to participate in system-level decision making and trades. 

e SE and SW life cycles, processes, and products are not always consistent or 
harmonized for meaningful cross-discipline integration to occur. 

¢ Segregation of SE/SW proposal inputs can impede coordination. 

¢ System development methods do not properly leverage SW ability to rapidly 
field enhanced capabilities. 


Recommendations: 
Institutionalize the integration and participation of software 
engineering in all system engineering activities. 

¢ Designate an empowered SW architect (acquirer and supplier) early. 

e Update policies, standards, guidance to emphasize SW in decision-making. 

e Promote effective SE/SW communications and cross-training. 

e Adopt acquisition strategies and development approaches that leverage rapid 

SW development cycles (iterative capability, sustainability). 
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Issue 3: Software life cycle planning and 
management by acquirers and suppliers is =. 
ineffective. (GTH THROUGH INDUSTRY & TECHNOLOGY 


TIONAL 
STREN 


Discussion Points: 

e SW risks and life cycle costs are not consistently accommodated in planning. 

e Realistic schedule and effort (cost) estimates are often rejected or constrained. 

e Pressure to rapidly procure new capabilities can inhibit balance of life cycle 
cost, schedule, performance expectations to achieve executable programs. 

e Lack of SW expertise applied in planning and management. 

e SW processes/methods improperly selected, tailored, executed, monitored. 

e SW measures are not used effectively or acted upon. 


Recommendations: 
Establish a culture of quantitative planning and management, using 
proven processes with collaborative decision-making across the 
software life cycle. 
e Establish collaborative contractual relationships enabling joint ownership of 
issues. 
e Change culture to base expectations on historical data and realistic constraints. 
e Emphasize selection of appropriate processes/methods and codify in contract. 
e Incentivize life-cycle perspective and iteration to circumvent near-term thinking. 
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Issue 4: The quantity and quality of domain-knowledgeable Tow 
software engineering expertise is insufficient to meet the rapt 
demands of the government and the defense industry. shat 


Discussion Points: 
¢ Insufficient highly skilled domain-knowledgeable SW professionals to meet the 
growing demand, due to inadequate funding, insufficient career incentives, 
competition, downsizing, etc. 
e Insufficient infrastructure to rapidly transition skilled staff to new programs. 
e Inadequate SW staffing across the system life cycle (architecture, trades, etc.) 
e Career path incentives are insufficient to attract SW professionals. 
Recommendations: 
Collaborate on innovative strategies to staff to appropriate levels 
and to attract, develop, and retain qualified talent to meet current 
and future software engineering needs in government and industry. 
¢ Develop a Software Competency Roadmap, with strategic decisions about the 
future workforce (competencies, downsizing, R&D, etc.) 
Establish joint government, industry, academia working group to identify 
strategies for expanding SW resource pool and skill sets. 
¢ Cross-train among functional disciplines (PM, SE, etc.) to expand SW 
engineering pool and extent of domain knowledge. 
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Issue 5: Traditional software verification techniques 
are costly and ineffective for dealing with the scale 
and complexity of modern systems. 


Discussion Points: 


¢ Over-reliance on testing alone rather than robust SW verification techniques. 
e Manual testing techniques are labor-intensive, scale poorly, and are 
unproductive relative to the large investment of resources. 
¢ Compliance-based tests do not adequately cover risks or failure conditions. 
¢ Tests are over-documented with disproportionate effort on detailed procedures. 
e Education, training, certifications are inadequate to develop effective test skills. 
Recommendations: 
Study current software verification practices in industry, and 
develop guidance and training to improve effectiveness in assuring 
product quality across the life cycle. 
e Sponsor a study of state-of-the-practice verification and testing approaches. 
e Review/update testing policies and guidance to emphasize robust, productive 
approaches that maximize ROI. 
Review adequacy of verification plans/approaches early in the acq. life cycle. 
e Emphasize skilled investigation throughout the life cycle, based on coverage, 
risk mitigation, high volume automation. 
e Strengthen curricula, training, certifications, career incentives for testing roles. 
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Issue 6: There is a failure to assure correct, 


i : 
predictable, safe, secure execution of complex ie 
software in distributed environments. 


Discussion Points: 
e SW is inherently vulnerable to widespread SW assurance threats — we must be 
confident in the supply chain pedigree. 
Current techniques are inadequate to verify assured components with well- 
understood properties. 
e Assurance of systems, and SoS, cannot be easily inferred from components 
due to issues such as composability, emergent behavior. 
e Exhaustive testing to rule out vulnerabilities is not feasible. 
Recommendations: 
Collaborate with industry to develop approaches, standards, and 
tools addressing system assurance issues throughout the 
acquisition life cycle and supply chain. 
e Establish collaborative initiatives to develop and deploy comprehensive system 
assurance approaches. 
- Resistance to intrusion and compromise 
- Commercial standards to address vulnerability throughout the supply chain 
e Define SW assurance quality attributes for addressing in architectural trades. 
e Sponsor system assurance research, policy, guidance, training.. 
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Issue 7: Inadequate attention is given to total lifecycle 
issues for COTS/NDI impacts on lifecycle cost and 
risk. 


Discussion Points: 

e Inadequate estimating methods for COTS/NDI (reuse, open source, GOTS, .. 

¢ COTS/NDI best practices are known but not consistently implemented. 

e Inadequate attention to sustainment issues early in the life cycle. 

¢ Open source licensing issues can expose organizations to liability, loss of data 

rights, potentially large rework. 

¢ Customer expectations for customization reduce benefits of COTS solutions. 

¢ Insufficient infrastructure to create and facilitate reuse across organizations. 
Recommendations: 
Improve and expand guidelines for addressing total lifecycle 
COTS/NDI issues. 

e Encourage adoption of COTS/NDI best practices (SEI, AIAA, etc.) 

e Ensure COTS/NDI life cycle processes are addressed in program plans. 

e Review COTS/NDI life cycle support issues and tradeoffs at program 
milestones and reviews. 
Ensure COTS/NDI issues are addressed in OSD policy and SW Assurance 
initiative. 
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Summary Recommendations 
Top SW Issues 2006 


1. 


7. 


Enforce effective software requirements development and 
management practices, including assessment of change impacts, for 
both the acquirer and the supplier organizations. 

institutionalize the integration and participation of software 
engineering in all system engineering activities. 

Establish a culture of quantitative planning and management, using 
proven processes with collaborative decision-making across the 
software life cycle. 

Collaborate on innovative strategies to staff to appropriate levels, and 
to attract, develop, and retain qualified talent to meet current and 
future software engineering needs in government and industry. 

Study current software verification practices in industry, and develop 
guidance and training to improve effectiveness in assuring product 
quality across the life cycle. 

Collaborate with industry to develop approaches, standards, and tools 
addressing system assurance issues throughout the acquisition life 
cycle and supply chain. 

Improve and expand guidelines for addressing total lifecycle 
COTSI/NDI issues. 
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= Software Engineering Institute CMMI. 


CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= How to do them In One Fell Swoop 


Or: How do we do this in order to save 
time and money? 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 
= How to do them in one fell swoop 


Why do this? 


Many corporations are ISO Certified, but want 
to get to CMMI Maturity Level 3 for contracts 
and competitive advantage. Other companies 
want to see their products overseas, and ISO 
Certification is required 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Why do this? It can be done! (=) 
Without Fear of Disaster! 


There are many similarities between the CMMI- 
Dev 1.2 and ISO 9001. 


The book that can be used to look at the 
similarities and differences is: Systemic 
Process Improvement Using ISO 9001:2000 
and CMMI by Mutafelija and Stromberg 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


How do we go down this road and merge the 
requirements for the Standard and the Model? 
How do we make sure that we are not going 
down a blind tunnel without direction? 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


There are Prerequisites to performing 
an Appraisal/Audit in an organization: 
1) Education 


2) Application of the Necessary 
Knowledge 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Education: is the key: educate your 
folks! 


If you are ISO 9000 Certified you already have 
an internal audit group (8,2,2), also if you are 
CMMI Maturity Level 3, you are performing 
many of the practices in OPF. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Resources for Quality and Process 


ISO 9001 States: 


Section 8.2.2 Internal Audit, of the Standard reads “the 
organization shall conduct internal audits at planned 
intervals to determine whether the quality management 
system a) conforms to the quality management system 
requirements and b) is effectively implemented and 
maintained. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 
How to do them in one fell swoop 


=™ Resources for Audits 


Audits have to be planned, with the processes and 
areas to be audited identified, as well as the previous 
audit results. 


Actions have to be taken to answer the nonconformities 
found during the audits along with their causes. Audit 
criteria, scope, frequency and methods are to be 
defined. 

Name 2 Process Areas that relate to these 


activities. 


© 2005 by Carnegie Mellon University CMMI Overview Page 8 


Carnegie Mellon a . 
Software Engineering Institute CMMI. 


CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 
How to do them in one fell swoop 


= Activities for Process Improvement 


For OPF, the purpose is to ‘plan, implement, and deploy 
organizational process improvements based ona 
thorough understanding of the current strengths and 
weaknesses of the organization’s processes and 
process assets.’ 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 
How to do them in one fell swoop 


= Activities for Process Improvement 


In OPF, the responsibility for facilitating and managing 
the organization’s process improvement activities, 
including coordinating the participation of others, is 
typically assigned to a process group. The organization 
provides the long-term commitment and resources 
required to sponsor this group and to ensure the 


effective and timely deployment of the improvements 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 

Where am I going with this when | say 
‘Educate’? 


™ Understanding the Standard 


Ensure that those performing internal audits have had 
training on the ISO Standard and also understand how to 


audit 


= Understanding the Model 


Ensure that those who work with PPQA and OPF, should take 
an Intro to the CMMI class or equivalent. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


™ The Resources that are required for the 
ISO Standard can be the same 
resources required for the model. 


The resources (People, time, $)used for both the Standard 
and the Model can be combined to be the same group. 
Leverage the intellectual capital in your organization. Use 
the people’s brains!!!! Do not create two different groups 
to perform:the same function! CMMI Overview Page 12 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: An Example 


Quality Manual 


Requirements for a Quality Manual per Secti 
Standard: 


A Quality Manual must exist. It must contain all documented 
procedures that have been established to cover all the 
procedures for the Quality Management System (QMS). 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: An Example 


Quality Manual 


The manual covers the scope of the QMS, inciuaing aetalis 
and justification for any exclusions. 


The manual covers descriptions of the interactions between 
the processes of the QMS. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


™ Application: An Example 


Organizational Process 
Definition 


SG 1 Establish Organizational Process Assets 
SP 1.1 Establish Standard Processes 
SP 1.2 Establish Lifecycle Model Descriptions 
SP 1.3 Establish Tailoring Criteria and Guidelines 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: An Example 


Organizational Process 
Definition = Quality Manual 


If you have implemented OPD and/or have a Quality manual, 
you most likely have satisfied both the model and the 
standard. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: Bottom Line 
Organizational Process 
Definition = Quality Manual 


Do NOT produce another set of procedures for you ISO or 
CMMI effort if you ALREADY have procedure in place. 


Make what you have work for you. Do not do more work than 
you have to. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: An Example 


Training: Model Requirements 
for Organizational Training 


The purpose for OT is: to develop the skills and knowledge of 
people so they can perform their roles effectively and 
efficiently. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: An Example 


Informative material from CMMI v1.2, OT: 


An organizational training program involves the following: 


-Identifying the training needed by the organization 
-Obtaining and providing training to address those needs 
-Establishing and maintaining training capability 
-Establishing and maintaining training records 
-Assessing training effectiveness. 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: OT SGs and SPs 


SG 1 Establish an Organizational Training Capability 

-SP 1.1 Establish the Strategic Training Needs 

-SP 1.2 Determine Which Training Needs Are the 
Responsibility of the Organization 

-SP 1.3 Establish an Organizational Training Tactical Plan 

-SP 1.4 Establish Training Capability 

SG 2 Provide Necessary Training 

-SP 2.1 Deliver Training 

-SP 2.2 Establish Training Records 

-SP 2.3 Assess Training Effectiveness 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: 6.2.2 Competence, 
awareness and training 
Organization shall 


a) determine the necessary competence for personnel 
performing work affecting product quality (SP 1.1) 


b) provide training or take other actions to satisfy these needs 


(SP1.2, 1.4) 
c) evaluate the effectiveness of the actions taken (SP 2.3) 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 


= Application: 6.2.2 Competence, 
awareness and training 
Organization shall 
d) ensure the its personnel are aware of the relevance and 


importance of their activities and how they contribute to the 
achievement of the quality objectives, and (SP1.4) 


e) maintain appropriate records of education, training, skills 
and experience (SP2.2) 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 

= Now the answer you have all been 
waiting for: 


Appraisals: 

SCAMPI A,B,C? 

Internal Audit by your folks, or external 
by your Registrar? 

What are your needs? 
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= Now the answer you have all been 
waiting for: 
Appraisals/Audits: 


Where are you in your implementation? 


SCAMPI C: only one person necessary who can do the 
Internal ISO Audit as well as a B,C Team Leader 


SCAMPI B: a team of at least 4, all having the team 
requisites and all being internal ISO auditors 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 

= Now the answer you have all been 
waiting for: 
Appraisals/Audits: 


Where are you in your implementation? 


SCAMPI A: Of course a Lead Appraiser, who Is also an 
internal ISO auditor (at least), and one representative 
who is from the Registrar (having attended the 
approved SEI intro class). 
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CMMI® Version 1.2 Appraisals and ISO 
9001 Appraisals/Audits 
= How to do them in one fell swoop 


Example: 
A Tool to help you put it all 
together....... 
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Pittsburgh, PA 15213-3890 


The CMMI® Product Suite and 
International Standards 


Dave Kitson, Jeanie Kitson, Terry Rout and Pedro Sousa 


*KAVO — awa rite Tiiriney 


® CMMI is registered in the US Patent & Trademark Office by Carnegie Mellon University 


Sponsored by the U.S. Department of Defense 
© 2006 by Carnegie Mellon University 


This material is approved for public release. 
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Agenda 


Evolution of Capability Maturity Models® and ISO/IEC 
15504 


ISO/IEC 15504 Overview and Status 
CMMI Synergy with International Standards 
Work to be Done 


Future Directions 


® Capability Maturity Model is registered in the U.S. Patent and Trademark Office by Carnegie Mellon 
University 


Carnegie Mellon oo 
Software Engineering Institute cum ® 


Progression of CMMs® and 15504 -1 
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1991 1992 1993 1994 1995 1996 


® CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University 
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Progression of CMMs and 15504 -2 
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2003 2004 2005 2006 


2001 2002 


2007 2008 2009 2010 
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What EXACTLY is 15504? 


ISO/IEC 15504 is an international standard being 
progressed by WG10 - a working group established by 
ISO/IEC/ITC1 SC7. 


Prescribes requirements for process assessment; is a 
framework standard. 


Consists of 5 documents (15504-1, ..., 15504-5); note that 
all parts are now published. 


Not the same as ISO/IEC TR 15504 - this is a 9-part set of 
documents published as ISO Technical Reports in 1998. 
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Key Premise Underlying 15504 


Assessments of process capability will yield results 
which are comparable as long as: 
e the process assessment model satisfies 15504 
model compatibility requirements 
¢« the assessment process satisfies 15504 
assessment requirements 
e the assessment contexts are similar 


Software Engineering Institute CMMI 
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Evolution of 15504 Specificity 


1st generation (pre-Pilanesburg) - practices were HARD-WIRED into 
15504 (as a normative component - BPG) 


2nd generation (pre-IS) - process definitions were HARD-WIRED into 
15504 (as a normative component - 15504-2) 


3rd generation (IS) - compliance requirements for Process Reference 
Models (PRMs) HARD-WIRED into 15504 (as a normative component - 
15504-2): 
e PRMs will be externally specified 
¢ 12207 includes as an amendment a PRM encompassing 
TR 15504-2 process definitions (and more) 


Carnegie Mellon 
Software Engineering Institute 


Architecture for 15504 


Modelimethod 
developers 


ISO 15504-2 (rev) ISP 
Requirements 
for Conformity 
(Compatibility) 


determine applicabality of 


Process 
Measurement Assessment Reference 


Framework Model Mode! 


Requirements 
for Compliance : 
determine suitabilify af 

i 
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CMMI and 15504 


A-spec requirements relevant to international 
standards: 
“2.3 Reference Documents 
2.3.1 Applicable ISO/IEC documents, including 
ISO/IEC 12207 and ISO/IEC 15504.” 


“3.1 Development User Requirements 
3.1.6 The CMMI Product Suite shall be consistent 
and compatible with ISO/IEC 15504.” 
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Synergy With Selected International 
Standards -1 


Identify areas where there are opportunities for synergy 
between key international standards and the CMMI 
Product Suite 


Exploit these opportunities by developing appropriate 
work products and/or liaising with appropriate 
individuals and organizations 
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Synergy With Selected International 
Standards -2 


Two of the key areas identified to date are the ISO 
9000:2000 family of standards and the ISO/IEC 15504 
standard 


Note that 15504 provides a mechanism for 
establishing important relationships to other 
important process-related international standards 
such as ISO 9001:2000, 12207, 15288 
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Identification of suitable process reference models, 


Mappings between CMMI model (s) and PRM (s) 
satisfying 15504 -2 requirements, 


Translation mechanism 
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Translation Mechanism 


Purpose: for representing the results of an 
assessment as a Set of process attribute ratings 
for each process selected from the specified 
Process Reference Model 


How?: 


1. direct translation of Process Assessment Model 
ratings into a process profile 

2. or the conversion of the data collected during the 
assessment (with the possible inclusion of 
additional information) through further judgment 
on the part of the assessor. 
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Status 
A detailed mapping has been developed from CMMI 
to: 


¢ the Measurement Framework in ISO/IEC 15504; and 
¢« the Process Reference Model included in ISO/IEC 
12207. 


A possible approach has been developed for verifiable 
conversion of results. 


Te 
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An Approach for Data Conversion - 1 


For the Process Dimension: 
¢ Collect data and generate observations for all Specific 
Practices in Process Areas within the scope of the 
assessment. 


¢ Using the mapping tables, assign the data observations 
to all Process Outcomes and Attributes indicated from 
the mapping. 
- Thus, if a Specific Practice maps to multiple Process 
Outcomes, assign the data to all of the possibilities. 


¢e Review the assigned data and remove observations that 
are not relevant. 


e Judge whether the remaining data is adequate to enable 
judgement of achievement of the outcomes. 
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An Approach for Data Conversion - 2 


® 


Where adequate data is available, consolidate the data 
to the process level and rate the performance of the 
process (Process Attribute 1.1). 


If the data is not adequate, either: 
¢ Collect additional data; or 
e Remove the process from the 15504 scope of the 
appraisal. 
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An Approach for Data Conversion - 3 


Repeat the exercise for the Generic Practices and Process 
Attributes 2.1 — 5.2. 


Additional data will need to be collected to address gaps in 
the mapping 
¢ Most notably in Process Attribute 3.2 


In some cases, evidence from Advanced Practices may also 
be relevant to a particular Process Attribute for a specific 
Process or set of Processes. 


- ® 
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Tool Support 


To achieve this form of conversion is complex and 
difficult, without tool support. 


A tool (the Appraisal Assistant) has been developed 
for managing evidence and ratings in CMMI 
appraisals, with a specific focus on conversion of data 
to generate 15015504 Process Profiles. 


The tool automatically generates lists of data fora 
Process Attribute based upon detailed ratings tables. 
e It permits de-selection of data judged to be not 
relevant. 
e It highlights areas where there may be a lack of 
relevant evidence. 


te 
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The oe Assistant 


Copyright 2004 Software Quality Institute, Grif 
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Model Details 


Standard Appraisal Model Editor 


| tdi : - F \ 7 
Process Specific Dimension | Generic Dimension| Name Estimate the Scope of the Project tamework Selection 


@cmm 
+) [OT] Organizational Training Acronym Capability Level 1 ¥| ©1s012207/15504-5 
+) [OPF] Organizational Process Focu . — Hr 
f [OPD] Organizational Process Defir re Ceviy 
+) [OPP] Organizational Process Perfc ~, Feature PLL. Type 

[OID] Organizational Innovation ar 
=) [PP] Project Planning 

is). SG 1 Establish Estimates Appraisal Consideration 


Sonne - Determination of WBS usage for this practice must be based on 
=35P 1, 1-1 Estimate the Scope : 
| P i 
Tek descriptions topevel WBS only, notits fully elaborated and expanded form as 


referenced in subsequent practices of this PA 
Work package descriptit -Topevel work breakdown structure should be driven by and linked 
Work breakdown struct = to specified product requirements, (See Requirements Management 
Topevel WBS revision PA) 
- SP 1,2-1 Establish Estimates 
(#)- SP 1,3-1 Define Project Life ¢ 
(#) SP 1,4-1 Determine Estimate Establish a toptevel work breakdown structure (WBS) to estimate 
SG 2 Develop a Project Plan the scope of the project 
SG 3 Obtain Commitment to the 
+) [PMC] Project Monitoring and Cont 
+) [SAM] Supplier Agreement Manage 
[IPM] Integrated Project Managem — 
# [RSKM] Risk Management 
{(IT] Integrated Teaming 


+ [QPM] Quantitative Project Manaa: Related ISO-12207 Process Outcomes/ISO-15504-2 Attribute Achievements 


+) [REQM] Requirements Managemen F311 Th Five work for th ects defred? : 
a : eed © SCOPE O' work for the project is defined(Process speci 
"3 RD] Requirements Development F,3.6/4 The reuse potential of each domain is assessed(Process specific 
[TS] Technical Solution F.3.6/5 Reuse proposals are evaluated to ensure the reuse product is 
[PI] Product Integration 
1 TIOMT Trtenented Chmnline Mansnane Be, 
ll | 


[¥] Practice Induded in Stage Representation Add Go 


—=— Carnegie Mellon 
—— Software Engineering Institute 


(2) 


Pilot Appraisal 


15504-conformant SCAMPI A appraisal conducted at 
Critical Software, SA during February-March 2006 


Maturity level 3 scope 
Process reference model embedded in ISO/IEC 12207 (amended) 


Mapping between CMMI and 12207 provided by Software Quality 
Institute 


Appraisal team included 3 SCAMPI Lead appraisers , one ISO 
9001 auditor and one SPICE assessor 


Support automation provided by Software Quality Institute - 
Appraisal Assistant 


Subset of 15504 process profiles generated as well as selected 
ISO 9001 outcomes. 
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Project Management Process 


Maps to elements from the following Process Areas: 
e Project Planning 
e Project Monitoring and Control 
e Supplier Agreement Management 
¢ Requirements Management 
e Integrated Project Management 
¢ Quantitative Project Management 


With minor linkages to 
¢e Organizational Process Performance 
e Risk Management 
¢ Verification 
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Project Management Process 


¥.3.1.3 Project management 
‘The purpose of the Project management process & to ent#y, establs, co-ordinate, and monitor the activites, tasks, and resources necessary for a project to produce a product and/or service. in the context of the project's requirements and constrants 


Fata reait 
rita . 4.t | PA2A | PA22) PA2.2 | PAI.2) PAS | PAS.2/ PAS.1 | PAS.2. Process Summary | ISO 15504 Profle Sumrrary 


JF ald 
F123 


a “ Process Attribute Achteverment : puly Aceeved 


System tesing : ; 
a ~ Syangmn : Weakness : 
Stem repurements.... 
System arontecturat 
Software design 
‘Software conseucton: 


Scttware ntegraton 
Outcome: 1) The seupe of Che work bor tre project i ae 


©) archtecture (cebverable) SIGAL PPS6 $-SP1.2-1 
© Auge Report WOWERS — REQM-SG I-SP1,5-1 
2 CSW.2004-PRL-1960-saterrammcetecres doc SMKE PP-S0 2-241 
© cow-opf- coun propect-formsioc OFF Prsg 2SP2.4-1 
(A CSW-QMS-2002-PCS-2256-preqnct fe cyewandt WOWERS PPSG YSPED1 
(2 CSWW-S1GA5-2005-Q4B-2007-ca-in cor SIGAL #56 1-S01,3-1 
2 CSW-SMICE-2004-0PL-2912-setemas-mice-cetaiad-pimn.mop SMKE PP-SG 2-5P2.4-1 
Detaled Plan Anstyseohese Gsse2 PP-SG 2-5P2.4-1 
DB Detated Plan Modetng Phase Gsse2 PPS 2572.41 
(2) Enterprne Architect fie (with uted with Raquest - fram Snancsyiterns - foe requinments change management) WOWEBS =—- REQMSG 1-5P1.5-1 
& Exceptions and non-<complances Gss2 REQUASG L.SPt 1 
& Ocramai Meeting mnute wen consortium GssP2 REQM-5G 1-5PL.5-4 
FAR Support Category WOWEBS —PP-SG 2.5P2,2-1 
fe) France! mans SIGAL PRSG 25P2.4-1 
& France! Proposal s1GAt PSG 2-SP2,1+1 
© perwi//nta.creca.ou orretien/ managament/orotect-managament/COW-QNS-2003-PCS-1770-sraject-management.og? G55P2 PP-SG 2-SP2.4-1 
E& bitenriew OPF Tearrs OFF REQN-SG 1.5Pt 5-1 
1) mrervew Pr ossez PP50 15134 
& Prervew SRICE toa SME PPSG 1-SP1.31 
[2] Management proposal SIGA PPSG 1-S01,3-1 
© Maser Pan Gsse2 PP-SG 2-SP2.4-1 
Meeting mnutes (Technica!) OPF REQU-SG 1-5P1.5-1 
© Modeling Overnmers Sresentanon Gsse2 PRSG 1572.27 
H) coen-crajact aetemasmice. doe SME PP-SG 2-SP24-1 
Le) OFF Tear Intencew OnF PP-SG 1-SPRI-d 
By Project Ufe Cycle Management Process Gssp2 PP-SG 1-SPA.3-1 
© Project Progress Reports SIGAL REQN-SG 1-5PL5-1 
© Proposat partbtechnicatproposst GssP2 PPSG 1-5P1,.21 
Prototype oresentanon to the cemtomer KE REQM-SG 1-SPE.S4 
(2) Requirements doubts meeting SMKE REQMSG 1-55.51 
REQM-SG I-SPL5-1 
© SIGAL~ Master Plan spp PSO 2-SP2.261 
1B statement of work 2 PSG 15P2,.31 
[e} System coquanements review repo REQN 5G $-5P1.5-3 
© technica! proposal 9-56 1-S04,2+1 
© usar requeements rev report PEQU-SG 1-SP15-1 
BD Workshop with customer for presefiting requirements for salyses REQM-SG 1-57.51 
© Wwowers Pm intervew PP-SG 1SPL.21 
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PA 2.1 — Process Management 


F214 Acqusson pr 
The pvooes of the Keavetion prenaration case to estable Wie sends and aya of tie acoution 2nG to communiati theva wth the ootentlalogtent 


‘Suppber selection 


Suppher montoeng 
° [PAZ | 4 
Qumorrar acceptance ALL PA2.2) PAS.1 PA3.2/ PAS.1 | PA4.2) PAS.1 PAS.2/ Process Surmmury ISO 15504 Profte Surrerary 


Requremsets etctanon Process Attribute Achievement t Fuly Achiewed 


Sytem testing 

iaumn vaabnia Strengt Weainess 
System requrements... 
Sytem moncectural ... 
Software requeeran.. 
Software desgn 
Software construction 
Software ntegrton 


Achievement: a) Objectives Jor Tyr performance of ive process arm xdentitiedt 


‘See WPA-300 and WPS-300 “Softvare Hegurements Engneamng”. Ths ¢ the panned 
‘tamk anc affort for performing the process, 


[) Engneerng councl masting mutes - TSH 2-6P2,10 
1) FAR Process Management Category . SAN-GG 2.67 2.1 
© Maser Pun TS-06 2.67 2.2 
(2) PHC Interview PH SIMICE TSG 2.67 2.2 
&) Project Pan TS-66 2-GP 2.2 
© Quality couned meeting menses . TS06 2-72.10 
) Quatty Manual i TS-6G BGP 2.5 
Preblem resolzton m... [2] Quaity Poknes . SAMGG 2467 2.1 
Organisational #igters... [2] SW Consructon Process we TSG 2-GP 2.2 
Project management © SW Design Process a. TSG 2-GP 2.2 


Reset Evdunces Delete Unchecked 


Achievement: b) Performance of the process is planned and monitored 


Hunan Resource Pro. Report on aude of the orpect conducted ae obnned. 
Hurran resource man... 


Tanne fe) Aude Report « 2006 SAM.GG 2-68 2.9 
[2 Audt Report Jan 06: SAM-GG 2-60 2.9 
& Cow-SIGAI.2005-Qap-2002-42-nian toc SAM-GG 2-6? 2.9 
 Oetates propect ptr TS-06 2GP 22 
[2] Document Review Records TSG 2-6P 2.8 
fe] terenview Pie GSSP2 SAM-GG 2-60 2.9 
[2] Ptermew OM SIGAI SAM-GG 2-GP 2.9 
>) master Pon TSG6 2.6 2.2 
FP] Meetng minutes (Technica) S06 267 2.8 
[2] mieszone SQA mport TS-GG 2.67 2.8 
B POR eniestona report TS-GG 2-GP 2.8 
(2) POR product assurance report 15-06 267 2.8 
[2] Prac Intereiewy PM SMICE 15466 2.6P 2.2 
[Y) Progect Detaled Plan TS-66 26P 2.8 
(2) Project Plan TS-G6 2-GP 2.2 
(2) Project SQA Plan SAMGG 2-67 2.9 
{¥) Review records TSG 2-467 2.8 
[e) SW Construction Process «TSG 2G 2.2 
EB) SW Design Process Organgsto... TSG 2-GP 22 
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PA 2.2 —- Work Product Management 


F.1.1.1 Acqustion preparation 

Derpose of the Acquation preparaton process & to extableh Ue reeds and gov’ of the acquetun and to Conmruncace tiese wah the Dotentey wopieTs 
Sepoier Hekction 
‘Supplier monKorng 
(Cumomner acceptance 
Supper tendenng = : 
Product relnase erent attribute 
Product acceptance &... ‘ 
Srmarects @hatatho Process Attribute Achievement © Fuly Actuved 
System testing ‘ 
Sofware nstateton 
Stem requrenents.. 
Systam ateneacrry 


BAL | PA2.1| PAZ2 |RA3.1 | 6A3.2) PA4i PA4.2 | BAS.1 | PAS.2) Process Summary | ISO 15504 Profle Sumrrary 


Sottevare ntegaton 
Satan tarts AChinvETent: 3) Requirements TOF LTH! WORK products Of Iie process ire Welieradt 


System ntegton 


=] 

©& Document Review Records TSGG 2-67 2.6 
Configuration manage... & mtervew PM-orF TS-GG 24P 2.8 
Quay assurance: BD Ptervew PM-SIGAL TS-GG 247 2.8 
Verfcation & Maser Plan TEGG 2-4 2.2 
Vadenon &F Meetng rewtes (Tecrncal) TS-GG 2GP 25 
Jont review milestone SQA report TS-GG 2-GP 2.8 
Aude POR n@estone report TS-GG 2-47 2.8 
Problem resobition rm. POR product assurance reset 5-66 2.40 28 
Orpaesazional signi. (2) PMC Ertervew PM SMICE YS-GG 24P 22 
Project management © Project Oerated Plan 5-66 2467 2.8 
© Provect Pan 7S-GG 24P 22 
©) Progect olan (QMS 3.0) ~ TS6626P28 
© Revew rocores 1S-GG 2-GP 28 
© SW Conetnacten Process ~» TS66 246P 2.2 
FF Sw Beagn Process tH... TS-GG2GP 22 


Wurras Resource Pro. 
Murra resource tram... 
Tracing 

Krowlege manager... 
Asset management 
Reese program mana... 


(<3) 

& Detated project plan TS-GG 247 2.2 
& Wtervew Pm Gsse2 SAM-GG 245 26 
Inlensew PM SIGAT SAMGG 240F 2.6 
& nitervew PM SMICE SANGG 2 26 
© tntervew Pm-ore SAM-GG 2GP 2.6 
G& btervew Pe-wowess SAMGG 27 28 
& rrervew SIGAL Team 15-96 2.6 2.6 
EF Master Pan TSG 24P 22 
(2 PMC trterveew PM SMICE TS:-G6 26P 2.6 
[Hl Project Pan 15-66 2.67 22 
BSW Conatrecnen Process TSGG 2GP 2.2 
Fe) SW Design Process « TSG6G 26P 22 
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Pilot Appraisal Report 


Result considered a successful demonstration of 
ability to fulfill 15504 requirements and generate 
15504 process profile relative to a separate process 
reference model, 


Demonstrated criticality of adequate supporting 
automation 
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Pilot Observations 


Improvement opportunities were identified for support 
automation 


Sample of 12207 processes were rated and a sample 
of 9001 clauses were examined 


Observed consistency between CMMI findings and 
prior 9001 audit reports as well as prior SPICE 
assessments, given the expected progress of the 
improvement program at Critical. 


=— Carnegie Mellon = 
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Issues from the Pilot Appraisal | 


Model-related: 
¢ Deficiencies previously identified in the relationship 
between CMMI Generic Practices and the 15504 
Measurement Framework had a significant impact 
on assessment of Level 3 Process Attributes. 


Technology-related: 
¢ Significant issues in usability of the “translation” 
functions. 
¢ Need to incorporate the ability to capture additional 
evidence where needed. 
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Further effort 


Additional pilot appraisals are required to identify any 
additional issues and to raise confidence in the 
translation mechanism as well as the translation 
requirements of 15504. 


Ideally a combined CMMI / SPICE assessment using 
two independent teams should be undertaken. 


The ability to perform “bi-directional” translation 
would be useful (i.e. determine CMMI capabilities from 
a SPICE assessment) would be useful. 
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Beta-2 Release 


Significant improvements to the Evidence Registry and to 
Ratings Generation. 


CMMI — ISO 15504 mapping functions improved in line with 
findings from the Pilot Appraisal. 


Available for download from 
http:/Awww.sqi.qu.edu.au/AppraisalAssistant 


The SQI is committed to two further beta releases of the 
Appraisal Assistant in 2007. 
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Carnegie Mellon 
Software Engineering Institute 


Beta Test Programme 
Initial release 2005 

Beta 2 release 28 July 2006 

503 registered beta testers 


217 downloads of Beta 2 version 


— Carnegie Mellon 


Software Engineering Institute Fa 
More Information 


The Appraisal Assistant: 
¢ http://www.sqi.gu.edu.au/AppraisalAssistant 


Contact Details: 
¢ David Kitson dhk@sei.cmu.edu 
¢ Loretta Kitson kamoljk@mindspring.com 
¢ Terry Rout T.Rout@griffith.edu.au 
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@ History of 
Process 
Improvement 


Strategy of 
Transition 


@ Manage 
Transition 


@Use OID in CMMI 
Transition 


@Result of 
Transition 


History of Process Motorola Global Software Group 
Improvement 


® GSG was formed in 1991 


® Funded from Motorola 
(Oe) 


Livingston 
est.2000) 


® 14 Centers with +5000 
Engineering Talents 


CMMI 


La: 1098) 
 GSG China established in 
1993 


GSG China has 3 branch 
offices with 1200+ 


employees 
o00 © 
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Motorola Global Software Group History 


India China Russia Scotland Israel CMMLI5: 
Singapore 
Australia Poland, Argentina 
Malaysia CMMLI5: 
Singapore Russia 
Canada, CMM 35: 
Italy Argentina 


| 
CMMI5: 
CMM 5: China andia, Mala 2 
CMM 3: 
Florida, Arge 
CMM 5: 


CMM 3: 

Italy Poland 
CMM 4: Australia CMM 5: Illinois 

Singapore, Russia 

Malaysia, Canada 


CMMI 5: China 
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GSG China Software Improvement 


REBRPRRIRD on 


MI L5 
(SCAMPI A) 
2000 
We are the 
FIRST S/w 
organization 


199 in China 
Reassessment assessed SEI 
199 at CMM L4 CMML5 7 


Achieve 
199 CMM L4 


We are the 
first one in 
China who 
started to a 


practice mel 


CMM es 
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Motorola Process and Quality Initiatives 


— ol et Align and Link Individual Goals,Business Goals 
 — See)... and Motorola’s Strategic Goals 
Quality Records REEF... 


Capable Quality System 


me AY 
Sieuigma: 
Black Belt Inifiative, Apply DMAIC 


Process and Tools to Projects 


Technology 


am, 
CcCmmi 


Start Transition to CMMI From oe | 


Use QSR to Achieve Registration to ISO 9001/QS9000 
Standard 


EI Focus 
First SEI Assessment in 1980’s 


Motorola Document Classification, File Name, Rev Number 


Add additional legal text here if required by your local Legal Counsel. AA i leyge)-lo)m-\ 


MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. 
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005 


Sponsorship and Commitment 
Strategy of «@ Driving as one initiative linked to business goal 


Transition 
¢ Commitment and Involvement -+60% 


population direct involvement 


Managing the transition using project 
management process 


= Design with Reuse 
= Implementing CMMI incrementally 


= Automating Process environment to reduce 
process overhead 


® 


@ Manage 
Transition Project 


Transition Planning 
* Sponsor from Senior Management of CMMI 
adoption 


é Understand the business value of 
transition and setup as business goal 


* Establish Target Profile based on 
organization needs 


= Develop CMMI experts to drive the transition 


= Reinforce SEPG and re-organize SME team 
better support CMMI structure 


= Identify gaps using SCAMPI C 


® 


Defining the Target Profile 


Continuous Representation to 
focus on critical processes 


Target CMMI Profile 


54 eT 
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Q few 
IPPD & System Engineering Pideeds Areas 
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project nature 
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Transition Life Cycle 


Implementation Appraisal 
Process 
Redesign SCAMPI B 
Institutionalization 


Mass 
Deployment A Follow-up 
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Transition Teams 


Steering Committee 
—Project Sponsor 


Core Team to drive the transition 
-Project Management 


Involvement in process 
redesign and appraisal 
-Project Teams 


FAR Teams 
Working Groups Practitioners 


Subject Matter Experts -Project Team 


aaleitsicemy Member 
All Level 


Direct involvement of ~60% of the organization 
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Sample Transition Timeline* 


Transition Kick-off, Jan 2004 
CMMI training, Feb 2004 
Gap Analysis with SCAMPI C, Mar 2004 
Redesign Plan, Apr 2004 


Process redesign, May-Sept, 2004 
| Second SCAMPI GC, Sept 2004 


SCAMPI A, Sept 2005 


Institutionalization Aug 2005 
SCAMPI B, Mar, 2005 
Mass Deployment, Jan-Jun 2005 


Pilot, Oct-Dec 2004 


* Based on GSG-China data 
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Planning &Reg Analysis 
= Formal CMMI Transition Plan-Taking 
advantage from disciplined project management 


= Estimates, milestones, dependencies, risks etc. 


® Define transition process and setup quality 
control gate 


@ Manage Progress tracking within teams and by senior 
Transition management 


« Group team using PA categories. Each PA has 
PA 2 owners plus a support SME team. 


* Understand Transition Requirement 


“ Take SCAMPI C result as the input of requirement 
Take Target Profile as the input of requirement 
Process Opportunity Request Database 


® 


@ Manage 
Transition 


Process Design Strategy 


= Use OPD as guide to design Process and 
Process Asset 


= Develop template to provide the guide on how 
to develop process asset 


® Review and refine Process Architecture before 
process asset design 


= Design the process asset as an integrity system 
25 Howto present GP in OUR system 
2; How to design support PA in OUR system 
2; Interface design and control 


= Design with quick-win strategies ( fast, 
practical, easy, cheap, tool support, etc) 


= Inspections, SCM ,Audits & Quality control. 


® 


Process Design Focus 


= Engineering Process Improvement 


= Process integrity and process interface 
(eg how Support Pas support the Prj &Engineering Pas) 


High Maturity Pas (QPM, CAR, OID, etc) and 
the critical sub-processes (PP,RM,VER) 
@ Manage 


Transition Guideline on process tailoring and measuring 


= Tool Support planning synchronous with 
process design 


Using High Maturity Level Process Area 


Techniques &Tools in Digital Six Sigma 


UL nee ta Quantified Root Causes 
a Be “A h | 3 — 
wa : , 
= 


| 
HI cr 
P ; 


sis techniques 
2. Provide statibtical method 
to understand variation 


1. Provide analysis techniques 
2. Provide stat|stical method 
to understand variation 


SPC methodjand tool to 1. Quantitatively identify 
support baseline and model improvement opportunities 
establishment 2. Cost-benefit analysis 

3. Measure imjprovement effect 
by performante analysis 


Defects and 
other 
problems 


Project Data 


ealy SS800Jq woddns Jo bulsn 
CdIV SS8D0Jd |eAN7] UBIH Jo Bulsn 


Project Management & Engineering 
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GSG Process Measurement Environment 


Business Partner Environment 


GSG Metrics Database 


A ati Asset Characterization 
— PMBOOK 


Dev Team 
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Implementation 


Plan the training at the early phase and 
deliver training incremental based on role 


* Using OID as guide to transfer the process 
change into organization 


= Using OPF/POR system to track the process 
@ Manage change during transition 


Transition “© Monitor the overall change status and trend. 


= Assess and control the major change 
= Pilot, Controlled deployment and mass 


deployment based on the risk assessment and 
business cases. 


® 


Process Assets after Transition 


Current Asset Number 


“© Total 182 assets, including 33 
processes, 48 guidelines and 93 
templates & checklists. 


Policy Process Guideline Standard Template Checklist 


Asset Change in Transition 


«= Among the 47 new assets, 
there’re 6 processes. Others are 


Unchanged 


New 


. . 24% 
guidelines, templates and 
iz ape GB Unchanged 
checklists that better facilitate the m Changed 
implementation of our process. <a 
50% 
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@Use OID in CMMI 
Transition 


Pilot 


= Select pilot projects and prioritize the pilot 
package 


= Design the pilot package for pilot projects 
during different development phase to reduce 
pilot cycle time (product integrity and coverage) 


= Define Pilot successful criteria, goal and their 
measurement 


= Leverage pilot cost, cycle time, pilot coverage 
and quality 


* Formal plan and kickoff to get commitment 


* Timely pilot feedback via POR &SQE 
* Organizing SCAMPI C to check “Approach” 


® 


@Use OID in CMMI 
Transition 


Deployment 


Plan the training at the early phase and deliver 
training incremental 

é Role-Based 

é Focus on the CHANGE and new process 

é Interactive workshop vs “teaching” 


= Define different deployment approach to new 
project and “on the way” project 


= Develop deployment handbook to facilitate the 


usage to process 
éUnderstand process from perspective of life cycle 
éUnderstand process from perspective of PA 
& Use of process asset type 


= Process Handbook to build mapping between 
CMMI, Organization process and practice. 
= Manager review and Audit 


® 


Institutionalization 


= Upgrade new hire entry training to CMMI and 
SPP++ 
Plan the training at the early phase and deliver 
training incremental 

é Role-Based 

é Focus on the CHANGE and new process 

é Interactive workshop vs “teaching” 


= Define different deployment approach to new 
project and “on the way” project 


SUSS OID CMME = Develop and maintain PIIDs 


Transition 
= Use sample project as “role model” for sharing 


@Use OID in CMMI 
Transition 


Appraisal Planning 


= Plan SCAMPI C (2), B,C 


Using SCAMPI as a tool to identify the process 
improvement opportunities and milestones in 
transition 


= PIIDs development and maintenance paralleled 
with process development and deployment 


= SCM control and PIIDs Tool 
= SME Team focus on driving the deployment of 
high maturity practice especially in critical sub- 


process aredaS 


Good practice in critical process areas 
traceability monitoring at organization level 


® 


Controlling deployment via successive SCAMPIs 


Institution- 


Mass 


alization 


Deployment 


|| I {| || 


SCAMPI C SCAMPI C SCAMPI B SCAMPI A 


ut ” “Approach” “Deployment” “Institutionalization” 
Gaps” Check 
P Status Check Status Check Status Check 
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@Use OID in CMMI 
Transition 


Appraisal Planning 


= Sample Project Selection 
é Overall Coverage 
é Applicability 
é Representative from location, business, 
domain, project category, current phase,etc 


 Pre-site Preparation. 

= Overview of organization 

= Overview of Process System , Projects for 
Appraisal team 

= Demo of Process Environment &Tool for FAR 


team 


& Pre-site Document Review and data collection 
request tracking 


Lessons in Transition Project 


= Develop CMM expert and understand SCAMPI 
method and process at early phase 


= Develop, maintain PIIDs at the early phase. 


* Understand PIIDs and provide right 
information at first time 


= Multi-site team overhead could be reduced 
through more effective coordination. 


@Use OID in CMMI 
Transition 


Transition Statistics 


Overall Transition Effort 
(staff*months) 

Overall Duration 

System Engineeringin included the 
scope 

IPPD included in the scope 


> 


SAM included in the scope 

Sw Population during transition 
period 

# of sites appraised 

Number of PAs in SCAMPI C scope 
Number of PAs in SCAMPIB scope 
Number of PAs in SCAMPIA scope 


| Factors impacting data variance 
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CMMI Transition Approach Summary 


Elements of CMMI Transition Approach | Faster | Better | Cheaper 
Organizing transition like a project «| xe] x | 
Focus on key PAs during SCAMPI C&B «| | 


Pils preparation/update during SCAMPI C&B (up to 1.5 effort 
reduction) 


Early feedback on process deployment (SCAMPI C&B) & Removal 
of significant weaknesses before Class A 


Balanced team membership (GSG and Businesses) i awe 


Keeping Core Appraisal team throughout a series of SCAMPIs x x) 


Several rounds of Redesigned Process review Sl ge | og 
a a. 
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Major Improvement After 
CMMI 


processes were modified to focus more on 
critical processes aligned with business objectives 


= Using QPM,CAR,OID process and techniques 
to support critical process effectiveness 


& Shifted its focus on processes from DP to 
the broader in CMMI 


Use Enterprise Project Management System as a 
7 lal n while use 
Project Pamphlet as a supply of EPMS 


= The I . , particularly for 
Result of RM,UT,VAL now w provide more practical 
Transition guidelines, templates, techniques, and tools. 


® 


Major Improvement After 
CMMI 


= CAR and OID-based processes were used to 
improve VER process effectiveness without 
compromising quality in specific classes of cases. 


= The RM/RD process was improved. Late 
additions and changes to requirements are measured. 


é Setup Organizational Baseline to quantitative 
manage requirements volatility. 


éThe measured and estimated impact of changes 
led to new project re-plan, estimation, and a 
factual basis for negotiation with customers. 


@Result of 
Transition 


CMMI Performance Results of GSG China 


Metrics Trend Improvements 
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Conclusion 


W Transition to CMMI requires substantial effort and 
strong commitment 


Disciplined project management is one of important 
success factor for transition 


From GSG experience, the iterative CMMI 
transition path with successive application of various 
SCAMPI appraisal types is proved to be effective 
mechanism for identifying gaps and controlling the 
CMMI deployment 


@Result of 
Transition 


@ History of 
Process 
Improvement 


Strategy of 
Transition 


@ Manage 
Transition 


@Use OID in CMMI 
Transition 


@Result of 
Transition 
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Objectives Model = 
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- STEP 


Building a Bi-directional Bridge 
Between Software ThEory and Practice 


Benefit Types 
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Indicators/Metrics and SPI 3 plies 2 
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- STEP 


Building a Bi-directional Bridge 
Between Software ThEory and Practice 


Costs, Return & ROI 


Funding provided by B4-Step/Science Foundation of Ireland (SFI) 13 


- STEP 


Building a Bi-directional Bridge 
Between Software ThEory and Practice 


RO! 


Funding provided by B4-Step/Science Foundation of Ireland (SFI) 14 


| Bri 
Betwee i Software TREOry and Pris ctice 


How RSROI Applies to CMMI — ey - STEP | 
Level 2 Staged ) 
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How RSROI Applies to CMMI — 3 - STEP | 
Level 2 Staged (cont.) 
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How RSROI Applies to CMMI — 3 - STEP | 
Level 2 Staged (cont.) 
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CMMI Level 2 to Benefit Map 3 ft , 
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Meeting the Business Objectives 3 - STEP | 
of an Organization 
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Lower Time to Market 3B if Be 
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What Process Areas are linked to - STEP 
Business Lareane 


al Bridge 
Betwee ‘ otlaaate ThEory and Practice 


| 


Funding provided by B4-Step/Science Foundation of Ireland (SFI) Ball 


Prioritization of Implementation Re} ft , 
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A Recommended Order of 
Implementation 


- STEP 


Building a Bi-directional Bridge 
Between Software ThEory and Practice 


2-7 Configuration Management 6-3, 4-2, 4-1 

RE) Organizational Process Focus 7-1, 7-2, 7-3 

3-7 Organizational Process Definition 6-2, 7-1 

3-8 Organizational Training 6-1, 6-3 

ee) Integrated Project Management for IPPD 6-3, 4-1, 4-2, 4-3, 4-4 
3-11 Integrated Teaming 6-3, 4-1, 4-3, 4-2, 4-4, 6-2 
4-1 Organizational Process Performance 7-1, 7-2, 7-3, 6-2 

5-1 Organizational Innovation and Deployment 7-1, 7-2, 7-3 
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Questions? 3) pit a 
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Contacts — University of Limerick ey pb ae ” 


~— 


fionbarr@patmedia.net 


ita.richardson@ul.ie 
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‘Acquirer vs. Supplier Roles 


Key Acquirer Roles* 


e Business analysis/relationship 
management (incl. requirements) 
Leads ! nitors e Contract development & supplier 


management 
e Program ownership / project 
management 


Acquirer & Supplier Contractual Touch Points : Technical architecture, standards, 
security 


Key Supplier Roles 
Suppliers 


e Application Design/Development 
e System Maintenance 

e Desktop / Service / Help Desk 

e Hosting 

e Data center / mainframe 


NN 


* Based on Forrester, “Functions to Retain when Outsourcing” (July 2004) 


Minimum Set of Acquirer Practices 


Business | 
Need 


Transition 
Mgmt 


& Project 
Planning 
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nevelo Integrate | Deliver & 
P & Test | Maintain 


Initial CMMI for Acquistion: 
A Lean Model 


Customer Supplier 


Tios for Success 


“...three-years, and then align them 
with the monies available to 
continue to support technology. 
The acquisition strategy 
covers the acquisition 
approach, business 
considerations, 

supplier agreement 
types and support of 

the technology solutions. 
Each of these strategy 
elements offers a ...” 


Tip: Carefully 
calibrate 
strategic 
choices for 
the acquisition 
strategy 
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When CMMI-AM 
No Longer Suffices 
November 14, 2006 
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All opinions, conclusions, recommendations, 
either expressed or implied, are the speaker’s. 
They do not necessarily represent the views of 
the DoD, IRMC, NDU, or any other government 
organization. 
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Presentation Overview 


> Presentation is a two-person “tag team” presentation. 


> 1st part presents a general argument for a new integrative [CMMI] perspective. 
=» CMMIl-centered. 
= How the landscape has changed and the need for a more holistic approach 


> 2nd part presents an argument for smarter strategic sourcing (commercial 
definition) processes. 
» Strategic Sourcing-centered. 


= Expectations of order of magnitude change in sourcing efficiencies and effectiveness to 
survive, 


= Post contracting management focus over the contract itself as a driver of success, 
= How value realization overshadows cost management in a non-commodity environment, 
# Utilizing collaborative solutioning to outperform competitive forces, 


= Performing the critical balancing act of relationship management and tough negotiations to 
maximize success. 


CMMI Vision 


The initial vision for CMMI was to integrate 
the competing maturity models and 
provide more consistent process 
improvement 


¢ Cause integration of the functional disciplines 
within organizations and across programs 


¢ Increase systems engineering process 


maturity as organizations migrate from the 
sun-setting CMMs to CMMI 


= i ) . = j a ) H : " I : j s i i = t " / r : ; 5 f =z K C d ‘ 
Schaeffer, DoD Systems Engineering an , NDIA CMMI Conference, Denver, Nov 17, 2004. 


How we got where we are 


¢ CMMI Sponsors opted to pursue staged and 
continuous models to preserve legacy 
— SW-CM\M, staged 
— SECM, continuous 


We created “level-mania” instead of continuous improvement 


Have we lost sight of the goal? 


The end goal of CMMI is to provide a model for 
continuous process improvement, to achieve: 
— Reduced cycle times 
— Meeting cost & schedule targets 
— Improve quality 


When achieving a level replaces the focus on continuous 
improvement, we’ve lost sight of the goal 


_ >”. Level-mania 
SY The Solution 


DoD desires to shift focus from maturity levels to 

capability profiles 

— Remove the enticement of maturity levels and “one size 
fits all” syndrome 

Discourage use of maturity levels as selection 

criteria and replace with targeted CMMI-based risk 

and capability assessments & profiles 


Develop meaningful measures of process capability 
based not on a maturity level, e.g. Level 3, but on 
process performance 


Goal is to improve the impact of CMMI on program performance " 


¢ CMMI has the potential to do even greater 
things for development of Systems 

¢ “Continuous Improvement’ is just as 
important for the CMMI product suite as it is 
for organizations 


The Department would like to increase focus on 
understanding the capabilities of both our 
organizations and our industry partners, 
instead of merely achieving maturity levels 


16 
ES 
Schaeffer, DoD Systems Engineering and C , NDIA CMMI Conference, Denver, Nov 17, 2004. 


CMMI Workshop, Sept 7&8, 2005 


- The workshop addressed several significant aspects of utilizing 
CMMI in the DoD and federal acquisition process that have been 
troublesome, and developed recommendations that the CMMI 
Steering Group. and DoD or federal acquisition agencies can 
address. Some issues that were discussed include: 

— Background on how organizations approach CMMI appraisals and why 
— Use of Appraisal Disclosure Statement by acquiring organizations 

— Formal guide to CMMI Usage for DoD 

— Training for DoD Acquisition Organizations in the use of CMMI for DoD 


— Government lack of understanding of need for mature SE content and 
practice 


— Specifying or requiring CMMI in RFPs 
— CMMI Appraisal expiration date 


Schaeffer, Workshop and Summit on | Use in DoD Programs, September 7, 2 


Industry to DoD: 5+1=>0 


Workshop Findings 


Programs execute at lower maturity levels than their organizations 

have achieved and advertised 

- Appraisals use small samples—don't cover all projects 

« You can't judge a program without appraising it 

« How can an organization's level be for “Life” when people and 
processes change? 

* High-maturity practices are not consistently applied at the project 

level after contract award 


* Is the completeness of appraisal disclosure statements adequate? 
* Low-maturity acquirers and high-maturity suppliers 


Schaeffer, DoD and CMMI, NDIA, Nov 15, 5 


The New/(est) look at CMMI for ACQ and DEV 


> Inthe current marketplace, there are maturity models, standards, methodologies, and 
guidelines that can help an organization improve the way it does business. However, 
most available improvement approaches focus on a specific part of the business and 
do not take a systemic approach to the problems that most organizations are facing. 


> By focusing on improving one area of a business, these models have unfortunately 
perpetuated the stovepipes and barriers that exist in organizations. 


> Capability Maturity Model® Integration (CMMI®) provides an opportunity to avoid or 
eliminate these stovepipes and barriers through integrated models that transcend 
disciplines. [pg 3] 


> Since 1991, CMMs have been developed for a myriad of disciplines. Some of the most 
notable include models for systems engineering, software engineering, software 
acquisition, workforce management and development, and integrated product and 
process development. 


> Although these models have proved useful to many organizations in different 


industries, the use of multiple models has been problematic. i 3 
' IRAE 


uisition Organizati *A Preliminary Report 
06-SE-005. June 2006. 


Dodson, et al, Adapting Cc 
Improving Processes for Bette 


CMMI ACQ ASSESSMENT OF SITUATION 


> Although these models have proved useful to many organizations in 
different industries, the use of multiple models has been problematic. 


> Many organizations would like their improvement efforts to span 
different groups in their organizations. 


> However, the differences among these discipline-specific models 
used by each group, including their architecture, content, and 
approach, have limited these organizations’ ability to broaden their 
improvements successfully. 


> Further, applying multiple models that are not integrated within and 
across an organization is costly in terms of training, appraisals, and 


improvement activities. [pg 6] 
reliminary Report, IP tie 


-SE-005. June 2006. — 


Dodson, et al, Adapting CNV 
Improving Processes for Be 


uisition Organizatio 


CMMI ACQ BASIC ARCHITECTURE: 16 + 6 


> The initial draft CMMI-ACQ contains unique acquisition 
practices in six process area that cover solicitation and 
supplier agreement development, acquisitions management, 
acquisition requirements development, acquisitions technical 
solution, acquisition validation, and acquisition verification. 


> The six process areas are supplemented by 16 process areas 
that cover project management, organizational and support 
process areas. 


> These 16 processes are necessary but not sufficient to 
executing as a successful acquirer. [ACQ] [pg 8] 


Dodson, et al, Adapting CM 
Report, Improving Processes for Better 


i : A Preliminary 
U-2006-SE-005. June 2006. 


CMMI-ACQ’s TWO ‘IMPROVEMENT’ APPROACHES 


> Levels are used in CMMI to describe an evolutionary path 
recommended for an organization that wants to improve the 
processes it uses to acquire its products and services. Levels 
can also be the outcome of the rating activity of appraisals. 


> Appraisals can apply to organizations that comprise entire 
companies, or to smaller groups such as a small group of 
projects or a division within a company. [ACQ] 


> CMMI supports two improvement approaches. 


= The first utilizes what is called the continuous representation; the 
continuous representation is based on capability levels. 


= The second utilizes what is called the staged representation; the 
staged representation is based on maturity levels. [ACQ] [pg 19] 


Information Resources Management College 


Dodson, et al, Adapting CMMI 


Report, Improving Processes for Better Pro -2006-SE-005. June 2006. 
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In The Capability First Phase Second Phase ‘Final’ 
Beginning Maturity Model Integration Model Integration Phase ? 
Models 

MS 2167a SW-CMM + CMMI w/ 2 reps CMMI ACQ ONE 
Prescriptive SE-CMM + phased/continuous + CMMI DEV MODEL: 
Unidirectional SA-CMM + CMMI AM CMMI SVC CMMI 
Standards “ACQ” 


NONE MANY FEW 3-in-1 ONE ? 
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A New Paradigm 


Capability Maturity Model® Integration 
(CMMI®) Version 1.2 Overview 


SM CMM Integration, IDEAL, SCAMPI, and SEI are service marks of Carnegie Mellon University. 
® Capability Maturity Model, Capability Maturity Modeling, CMM, and CMMI are registered in the U.S. Patent and 
Trademark Office by Carnegie Mellon University. 


Sponsored by the U.S. Department of Defense 
© 2006 by Carnegie Mellon University 


This material is approved for public release. 


Stellar Solution: Constellations 


. . Pa 
Carnegie Mellon ’ . 
Software Engineering Institute saa 


The CMMI Framework 


The CMMI Framework is the structure that organizes the 
components used in generating models, training materials, and 
appraisal methods. 


The CMMI Product Suite is the full collection of models, training 
materials, and appraisal methods generated from the CMMI 
Framework. 


The components in the CMMI Framework are organized into 
groupings, called constellations, which facilitate construction of 
approved models. 


¢ During v1.2 development, CMMI-SE/SW/IPPD/SS was moved 
to the CMMI for Development (CMMI-DEV) constellation. 


¢ Two new constellations have been commissioned by CMMI 
Steering Group: 
- CMMI for Services (CMMI-SVC 
- CMMI for Acquisition (CMMI-ACQ) 


— | CarnegieMellon , 
“= Software Engineering Institute CMML 


Three Complementary Constellations 


CMMI-DEV CMMI-SVC 
provides guidance provides guidance for 
for managing, delivering services 

measuring, and within organizations and 


monitoring to external customers 
development 


processes 


16 Core Process Areas 
used in all 


CMMI-ACQ 


provides guidance 
to enable 
informed and 
decisive 
acquisition 
leadership 
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re ; : EXECUTIVE OFFICE OF THE PRESIDENT 
e/ OFFICE OF MANAGEMENT AND BUOGET 


2) 
WASHINGTON, D.C. 20503 


May 20, 2005 


MEMORANDUM FOR CHIEF ACQUISITION OFFICERS 
CHIEF FINANCIAL OFFICERS 
CHIEF INFORMATION OFFICERS 
A 
FROM: Clay Johnson IL 
Deputy Direetor for Management 


SUBJECT: Implementing Strategic Sourcing 


The federal government spends approximately $300 billion on goods and services cach 
year. and federal agencies are responsible for maximizing the value of each dollar spent, 
Therefore, agencies need to leverage spending to the maximum extent possible through strategic 
sourcing. Strategie sourcing is the co/laborative and strictired process of critically analyzing an 
organization’s spending and using this information to make business decisions about acquiring 
commodities and services more effectively and efficiently. This process helps agencies optimize 
performance, minimize price, increase achievement of socio-econonne acquisivion goals. 
evaluate total life cycle management costs, improve vendor access to business opportunities, and 
otherwise increase the value of each dollar spent. 


Each ageney’s Chief Acquisition Officer (CAO), Chief Financial Officer (CFO), and 
Chief Information Officer (CTO) are responsible for the overall development and implementation 
of the ageney strategic sourcing effort, which begins with a spend analysis and the identification 
of commodities for which strategic sourcing should be implemented. The CAO shall lead the 
CAQ/CFO/CIO development team and will take the following actions: 


1. Not later than October 1, 2005, the CAO shall identity no fewer than three commodities 
that could be purchased more effectively and efficiently through the application of strategic 
sourcing, exeluding software that could be purchased under the SmartBuy program. Agencies 
may include existing strategic sourcing efforts for this purpose. 

2 The CAO shall lead the collaborative development of an agency-wide strategic sourcing 
plan in coordination with the agency CFO, CIO, representatives from the agency's Office of 
Small and Disadvantaged Business Utilization, and other key stakeholders. as appropriate, The 
plan should reflect the application of sound program and project management principles, Ata 
minimum, the plan should include the following elements: 
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a. Strategic Sourcing Governance — A charter should be developed outlining the 
members, roles, responsibilities, and operations of an agency-wide Strategic Sourcing 
Council and any commodity councils to be formed. 


b. Strategic Soureing Goals and Objectives — The Strategic Sourcing Council should 
establish annual strategic sourcing goals and objectives, by fiscal year. These goals 
and objectives should include existing strategic sourcing efforts, as well as 
prioritizing new initiatives. In addition to cost and performance goals, any strategic 
sourcing plan must be balanced with socio-economic goals for small businesses, 
small disadvantaged businesses, women-owned small businesses, veteran-owned 
businesses, service-disabled veteran-owned businesses, HUB-Zone and preference 
programs (e.g., Javits-Wagner-O* Day), and others, as appropriate. 


e. Performance Measures — The agency Strategic Sourcing Council should establish 
agency-wide performance measures and reporting requirements in order to monitor 
and continuously improve the strategic sourcing program. 


d. Communications Strategy — The Strategic Sourcing Plan should also include a 
communication strategy that clearly conveys senior management’s commitment to the 
effort, describes the scope of the effort, and identifies any organizational changes. 
The communications strategy should also include steps to make agency employees 
aware of awarded strategic sourcing contracts and how they are to be used. 


e. Training Strategy — The plan should identify actions necessary to educate agency 
personnel to support effective and efficient strategic sourcing implementation and 
management. 


Procurement Policy (OFPP) regarding, at a minimum, reductions in the prices of goods and 
services, reductions in the cost of doing business, improvements in performance. and changes in 
achievement of socio-economic acquisition goals at the prime contract and, if possible, the 
subcontract level. Agencies shall develop methodologies for establishing baseline data and 
subsequent changes to this baseline and shall consistently apply this methodology throughout the 
strategic sourcing process. 


3 Beginning in January 2006, the CAO shall report annually to the Office of Federal 


Using information from the agency reports and other data sources, OF PP may identify 
several commodities that could be strategically sourced government-wide, and will establish an 
interagency structure for managing the acquisition of these commodities. 


To facilitate the development of a strategic sourcing community and build a subject 
matter expert network, agencies shall identify a strategic sourcing point of contact. Please 
submit the person’s name, title, telephone number, and e-mail address to Lesley Field at OFPP 
by July 1, 2005 (lfield@omb.cop.gov). 


Maximizing value for taxpayers is a top priority for OMB. and I look forward to working 
with the acquisition community on this important initiative. 
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This Acquisition Process model contains four types of service levels: 
[Marais, Global Sourcing in an Increasingly Outsourced Environment, SIM ITPWG, 2006.] 


SL4 


App Support 
Helo Desk 


SL1 SL1 SL1 


SL1: Component Level 
SL2: Interface / Handoff 
SL3: Collaborative 
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Contract Contract 
Initiative Contract Implementation 
Start Signing Completed 


Contracting 
Efficiency 


¢—___. 


«—— Implementation —— 


Effectiveness 
Negotiation 
Effectiveness 


«—_____ Issue/Dispute Effectiveness ————_>* 


IT Strategy/Intent Achievement 
Contract Content Effectiveness 
Best-In-Class Benchmarking 


Level of Capability Implementation 


State of Strategic Relationshi IRAC 


Marais, Global Sourcing in an Increasingly Outsourced Environment, SIM ITPWG, 2006. 
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> Acquisition effectiveness is judged by what you measure 
> Acquisition process models should measure the necessary elements 
bilaterally in each of the following categories: 
= Partnership 
= Process 
= Continuous Improvement 
= Performance 
> Acquisition process models should differentiate between tough 
negotiations and good relationships which are not mutually exclusive 
> “Where’s the Meat?” 
\ ae acquisition processes depend on the bridging processes in the 
mi e. 
= Acquisition processes which focus on the two halves of the bun miss the 
meat. 
> One Unified Acquisition Process model for Acquirer and Provider 
solves the “gap in the middle” problem. 


Modified from Marais, Global Sourcing in an Increasingly Outsourced Environment, SIM ITPWG, 2006. 
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CMMI Acquisition Module 


¢ New CMMI Acquisition Module (CMMI-AM) 
should prove valuable in assisting program 
offices in improving acquisition process 

— Recent pilot efforts indicate positive effect 
Results not expressed as “Levels” but as 
Capability Profile 

Self-initiated, for internal use 

Will help put program offices on path to 
acquisition process improvement 


{2 
Schaeffer, 2004 SS ——~SC TINS 


Strategic Sourcing: Acquisition as an 
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> Industry Best Practices in “strategic sourcing” 
[might] represent a useful approach to consider for 
framing [future] integration efforts within the CMMI 
model “family” 
= The conceptual view of one CMM model for each 
end of a dumbbell (CMMI ACQ + CMMI DEV) 
might be found to work less well than one 
integrated model for whole process: 
© Focus on acquirer processes [one end] 
© Focus on supplier processes [one end] 
° Focus on acquirer-supplier processes 
[bidirectional evaluative gap-bridging 
processes] in the middle 


ACQUIRER ACQUISITION PROCESS MGT SUPPLIER 
processes €5 processes 
THIS IS BEST DESCRIBED and 


MANAGED within ONE MODEL 
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Speaker Biographies 


> Claude Marais is a seasoned sourcing and technology executive with more 
than 20 years global experience in major corporations such as General Motors 
and Coca-Cola. His background includes application development, global 
sourcing and outsourcing management with budgets of up to $3 Billion per 
year. As a thought leader in sourcing management he has played key roles in 
industry organizations. Amongst others, he is a Fellow and Honorary Vice- 
President of the International Association for Contract and Commercial 
Management and the founder of the Society for Information Management IT 
Procurement Working Groups. 


> Dr. Carl Clavadetscher is Professor of Systems Management at the Information 
Resources Management College (IRMC), National Defense University (NDU). Dr. 
Clavadetscher currently heads the acquisition portion of the IRMC Advanced 
Management Program (AMP) and IRMC ClO program. Prior to joining IRMC, Dr. 
Clavadetscher was Professor of Information Systems and Chair of the 
Computer Information Systems Department at Cal Poly Pomona, a department 
DPMA recognized as the nation's best (1988). Dr. Clavadetscher has held 
appointments at IBM's Los Angeles Research Center, Los Angeles Water and 
Power MIS, and the Rockwell Corporation. He is a combat decorated veteran of 
Vietnam (MACV). Dr. Clavadetscher has been an active speaker and author on 
IT issues for almost 30 years. 
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Raytheon 
Customer Succes. 


5 Is Our Mission 


No Doubt - The LRAS3/FS3 
Story of 
Mission Assurance 


Jill Brooks 
Sheleen Johnson 
Network Centric Systems 


Raytheon 
Agenda 

e Introduction to Raytheon 

e LRAS3/FS3 Overview 

e Sound Engineering Processes 

e Partnership With The Customer 

e Results Speak For Themselves 

e Summary 
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Raytheon 
Introduction to Raytheon 


¢ Raytheon is an industry leader in defense and government electronics, 
space, information technology, technical services, and business aviation 
and special mission aircraft 


e Network Centric Systems (NCS) develops and produces mission 
solutions for networking, command and control, battlespace awareness, 
and air traffic management 
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Raytheon 
Introduction to Raytheon —- continued 


Multi-site 


SW Level 5 
SE Level 3 


Level 3 Level 4 
10x Baseline 


Fault Density Validation 
Improvement 


Level 3 


- a aca Raytheon Seana otic ta ar adie a aca oe 


Acquisition Level 4/5 
eae Level 3 & Transitio Managed/ 
panere Defined Optimizing 
Repeatable 


IPI CMM.-based Internal Process Improvement Assessment 
RTIS_ Raytheon/TI Systems 


CMMI CMM Integrated 
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Raytheon 
LRAS3/FS3 Overview 


e The LRAS3/FS3 Is a long range reconnaissance and 
surveillance multi-sensor suite system with the capability to 
determine far target location (FTL) coordinates, and to 
provide real-time target detection, recognition, and 
identification capability to the scout while permitting 24-hour 
adverse weather operations. In addition, the FS3 variant 
provides laser designation of a target for laser guided 
weapons. 


LRAS3/FS3 Product supports scout and fire support missions 
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Raytheon 
LRAS3/FS3 Overview — continued 


e FS3 product supports Knight fire support missions. The 
LRAS3 product supports scout operations and is operable in 
both a stationary vehicle mounted configuration and in an 
autonomous dismounted configuration The host vehicle for 
the system is the High Mobility Multi-purpose Wheeled 
Vehicle (HMMWYV) and Stryker Armored Vehicle. 
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Raytheon 
Sound Engineering Processes 


e SW engineering processes assessed as CMMI Level 5, in 
2003 


e Rigorous approach to engineering practices, key examples 
include: 
— Requirements Management 
e Used DOORs tool to support bi-directional traceability 
e Ensures that the impact of changes to requirements is understood and 
addressed through out the system 
— Configuration Management 


e Used through out the life-cycle of the product from Requirements Definition, 
through design, implementation, test and maintenance 


e¢ A multi-tiered approach that has different requirements for different artifacts 
and for different points in an artifact’s lifecycle 


Engineering process supports Mission Assurance 
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Sound Engineering Processes — Raytheon 
continued 


e Additional engineering practices that played a key role in the 
mission assurance success of LRAS3/FS3: 
— Peer Reviews 
e Not the “glamorous” part of the process, but rigorous application pays off 
e Verification by Quality Engineering 
— Verification, Validation 
e An iterative approach 
e Back to basics, back to the requirements 
— Quality Engineering 
e Maintained independence, but an integral part of the program 
e Ensured adherence to defined processes 
¢ Worked with program to find solutions 


Engineering process supports Mission Assurance 
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Raytheon 
Partnership with the Customer 


e A series of demonstrations and user juries were conducted to 
refine the Raytheon understanding of the customer 
requirements: 


— User Jury #1 
¢ Combined with PDR (Preliminary Design Review) to ensure requirements 
are understood 


— User Jury #2 
e Paper copies of the display screens 
e End user participation requested (several different ranks) 


Demonstrations and User Juries were key tactics 
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Partnership with the Customer — 
continued 


Demonstrations and User Juries were key tactics 
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Partnership with the Customer — Raytheon 
continued 


e Demonstrations and user juries (continued): 


— Engineering Confidence Test: Demonstration #1 
e Fire LDM (Laser Designator Module) from external run box 
e External mounted camera - alleviate delays for packaging/fabrication 
e Insert filters in front of camera to simulate end system performance 


— Program Gate: Demonstration #2 
e Interim demonstration of more integrated system at longer ranges 


— Tactical Demonstration 
e After completion of the Vehicle Sensor Mount and end system development 


Demonstrations and User Juries were key tactics 
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Partnership with the Customer — Raytheon 
continued 


e Customer participation in major reviews 


— PDR — Preliminary Design Review 
— CDR -— Critical Design Review 

— TRR — Test Readiness Review 

— IPR — Interim Progress Review 


— All of these reviews involved customer participation and an agreement 
regarding actions and authorization to proceed 


Formal Reviews were a key tactic 
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Partnership with the Customer — Raytheon 
continued 


e Field Service Representative in place to facilitate 
maintenance issues for deployed systems. 


Maintaining systems assumes high priority. 
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Raytheon 
Results Speak for Themselves 


e Software CPI has consistently stayed within organizational 
limits for several years 


—@© CPI FS3 Dev (2002-2003) 
—®- CPI BB1 Insert (2004-2005) 
~~ Org Norm 1 

—<- Org Norm High 


Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar 
Achieved Budget as well as Mission Assurance 
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Results Speak for Themselves — Raytheon 
continued 


e Software SPI has consistently stayed within organizational 
limits for several years 


—@ SPI FS3 Dev (2002-2003) 
-a- SPI BB1 Insert (2004-2005) 
1.45 ~— Org Norm Low 

—<— Org Norm High 


SS 


Achieved Schedule as well as Mission Assurance 
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Results Speak for Themselves — Raytheon 
continued 


e Defect containment metrics for software remained within 
organizational limits over multiple development efforts 


120 —e— FS3 Dev (2002-2003) 
—s— BB1 Insert (2004-2005) 
—»— Org Norm Low 


—»<— Org Norm High 
100 = bd © \ % ca \ 
80 


; a are 
cd 7 te - 7 te * * * * te * * & * 


Oct | Nov | Dec | Jan | Feb | Mar | Apr | ae Jun | Jul | Aug | Sep | Oct | Nov | Dec | Jan | Feb | Mar | 
No Doubt — A Mission Assurance Success Story 
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Results Speak for Themselves - 
continued 


¢ To date, there have been two follow-on software 
maintenance contracts in addition to development effort to 
enhance software and hardware 


No Doubt — A Mission Assurance Success Story 


11/15/2006 | Page 17 


Results Speak for Themselves — Raytheon 
continued 


e Positive feedback from in-theatre user community 


— Highly positive feedback in multiple 
applications 


— Not originally designed for mounting on 
buildings, but modified and used on 
rooftops to monitor borders / 
surveillance 


No Doubt — A Mission Assurance Success Story 
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Results Speak for Themselves — Raytheon 
continued 


No Doubt —- A Mission Assurance Success Story 
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Raytheon 
Summary 


e Sound Engineering Processes coupled with a partnership 
with the Customer have led to success for the LRAS3/FS3 
program, Raytheon, and the Army: 


— A program that is within budget and on time 


— A product that performs — with no doubt 


No Doubt —- LRAS3/FS3: A Mission Assurance Success Story 
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Raytheon 
Contact Information 


¢ Jill Brooks 
— 972-344-3022 
— Jill A Brooks@raytheon.com 
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Raytheon 
Questions 


11/15/2006 | Page 22 


NORTHROP GRUMMAN 


DEFINING THE FUTURE 


“ag Using Voice of Customer 
»to Advance OID - 


* a Giving The Customer a 
ner NS 


\ _ Vote !n Improvement 


Le _ CMMI Technology Conference 
yh |DY=\a\Vic) mm @xe) (0) ¢-l0l0) 

7 Ar 

3 


nd _ November 11, 2006 
nse 


DYoyay-\(oM nem Oxelge)aeyal 
Division Manager — Six Sigma 
NYo)adalge)emeiaulanlant-lamene)sele)e-lulela 


Preview... 


Intent of Organizational 
Innovation and 
Deployment 


Definition of “Voice of 
Customer” 


Means to Gather Voice of 
Customer 


Translating VoC into 
Criteria to Select 
Improvement Projects 
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About Organizational I nnovation and 
Deployment... 


Quality and process- 
performance objectives that this 
process area might address 
include the following: 


= Improved product quality (e.g., 
functionality, performance) 


= Increased productivity 
= Decreased cycle time 


Greater customer and end-user 
satisfaction 


= Shorter development or 
production time to change 
functionality, add features, or 
adapt to new technologies 


NORTHROP GRUMMAN 
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‘VIoice of Customer’ is... 


“Voice of the Customer” 
(VoC) means to 
describe the wants and 
needs of our Customers 
and their perceptions of 
quality 


NORTHROP GRUMMAN 
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VoC is important because... 


= High maturity organizations 
recognize the tie between high 
quality and Customer loyalty, 
referenceability and it’s effect 
on business 


Quality is all non-price attributes 
of a product or service, tangible 
and intangible 


Only Customers define quality; 
should hear what they have to 
say 


Organizations need process 
improvements that align to 
“quality that Customers can 

see” NORTHROP GRUMMAN 
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Understanding quality ts difficult... 


= Customers don’t know all of 
their needs, or can't 
articulate what they are, or 
assume that you know them 


= Customers often state their 
needs in fuzzy terms like, 
“Easy to Use” 


=" Customers needs and 
wants frequently change 


= Quality is perceived, and 
not quantitatively measured 
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Follow six steps... 


@ Discover Their a © Choose 
Needs with Interviews © Prioritize Needs Improvements 
Based On Criteria 
® Identify ® Categorize © Survey to 
Customers/ Needs Determine What 
end-users Really Matters 
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Gather Voice of Customer with Reactive 
or Proactive Systems... 


Two basic types of VoC 
systems exist... 


= Reactive Systems 
= Proactive Systems 
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Reactive systems are... 


1@#%& 


Reactive systems are 
those where Customer 
information comes to 
you whether you want it 
or not, @.g., 

= Customer complaints 

= Award Fee Letters 

= Help Desk Calls 


= Contractor Performance 
Assessment Reports 


NORTHROP GRUMMAN 
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Proactive systems are... 


Proactive systems are 
those where we seek out 
information about 
Customers needs and 
wants, e.g., 


Interviews 
Surveys 

Focus Groups 
Executive Visits 
Benchmarking 
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Customer interviews are good to discover 
needs... 


= Interviews are proactive 
systems to discover 
Customers discover 
needs or wants 


Capture verbatim 
comments that describe 
needs or wants (on 
index cards or Post-It® 
notes) 


NORTHROP GRUMMAN 
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Record Customers’ verbatim comments on 
cards... 


For capturing 
comments... 


“= Cards should have 
independent thoughts or 


Give it to me —_ 
= Write large enough so 
faster that they can be seen 


drc from afar 
= Put initials on the card 


NORTHROP GRUMMAN 
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The approach to interviews... 


= Provoke images and 
listen 


= Employ open ended 
questions that don't 
imply a right or wrong 
answer 

= Tailor questions to 
the situation 


NORTHROP GRUMMAN 
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Find out about Customers’ current 
approach... 


How do you use the 
current system? 


Tell us about the last few 
times. 


How do you get service? 


Who or what provides the 
service? 


When ts the service 
available? 


How much time to 
respond? 


What method to respond? 


NORTHROP GRUMMAN 


Copyright 2006 Northrop Grumman Corporation 


Explore what is working well... 


Tell us what works well 
with the current 
approach? 


«= What not to change 
= What advantages now 


NORTHROP GRUMMAN 
14 


Copyright 2006 Northrop Grumman Corporation 


15 


Probe problem areas... 


Tell us about problems you 
have with the current 
system... 


Tell us about problems. 


Which most important to fix? 
Why? 


Any work-arounds? 


Any costs and negative 
consequences? 
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Ask about the ideal solution... 


Describe what you see 
as an ideal system... 


= What would it look like? 


=" How would you interact 
with it? 


= Any other systems 
you've seen or heard 
about it’? 


= Who does it best? 


NORTHROP GRUMMAN 
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Explore the future... 


17 


What changes do you 
anticipate in the next 
couple of years? Include 
changes in... 


Technology 

Your work 
Document types 
Presentation types 
Delivery formats 


Client demands or 
expectations 


World events 
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Needs Customers might tell you... 


“| want fast and 
easy access to 
help.” 


“Help me 
save time.” 


“If it breaks, 
fix it fast.” 


“| want it 
when | want 
ies 


“Make 
commitments that 
meet my needs.” 


“Meet your 
commitments.” 


NORTHROP GRUMMAN 
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More needs Customers might tell you... 


“Deliver 
irresistible 
value.” 


“Help me 
save money.” 


“Treat me like 
you want my 
business.” 


NORTHROP GRUMMAN 
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Even more needs Customers might tell you... 


“Give me systems 
that meet my 
needs.” 


“Be accurate, 


right the first 
time.” 


“Give me 
systems that 
work right.” 


“Fix it right 
the first time.” 


“Make it easy 
to use.” 


NORTHROP GRUMMAN 
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About affinitizing needs or attributes... 


21 


= The affinity diagram, or KJ 


method is one of the most 
widely used of the Japanese 
management and planning 
tools 


The affinity diagram was 
developed to discover 
meaningful groups of ideas 
within a raw list 


Important to let the groupings 
emerge naturally, using the 
right side of the brain, rather 
than according to preordained 
categories 

NORTHROP GRUMMAN 
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Start by posting all the verbatim 
comments... 


Team sorts cards into groups with a 
common thread... 


Do this as a 
silent activity 
Anybody can 
move any card 
any number of 
times 


NORTHROP GRUMMAN 


Copyright 2006 Northrop Grumman Corporation 


Someone works with the team to 
paraphrase the essence... 


Team 
paraphrases the 
main theme of 
each group 


NORTHROP GRUMMAN 
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About surveys... 


=" Surveys are another 
proactive system to 
collect information about 
Customer needs 


= Combined with Likert 
scaling, Surveys can 
prioritize needs 
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A Likert scale is... 


A Likert scale is a type of 
survey question where 
respondents rate the level at 
which they agree or disagree 
with a given statement, e.g. 


The system is easy to use. 


L 
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Likert scales are effective to... 


= Measure “soft” things like 
perceptions, attitudes, or 
satisfaction 


= Capture the state of 
information about process 
performance where only 
notions of performance exist. 


= Verify Customer's perception 
of differences in “before” and 
“after” 


= Validate that “hard” measures 
are good predictors of what 
Customers’ see 


NORTHROP GRUMMAN 
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Especially in software... 


In software evaluation, 
we can often objectively 
measure efficiency and 
effectiveness with 
performance metrics such 
as time taken or errors 
made 


But, Likert scales and 
other attitudinal scales 
help get at the emotional 
and preferential 
responses people have to 
the design 


NORTHROP GRUMMAN 
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Steps to survey... 


@ Compose ® Analyze with 


questions from logistic regression 
Interviews 

@ Determine 

the “Money @ Administer 

Question” the survey 


NORTHROP GRUMMAN 
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Determine the “Money Question”... 


30 


The “Money Question” is a 
question that addresses the 
overall trait of interest, e.g., 


“Overall, | am satisfied with 
the systems and services 
that Northrop Grumman 
delivers” 


“Overall, this training will 
help me do my job better” 


“Overall, | am satisfied with 
Northrop Grumman as a 
place to work” 


NORTHROP GRUMMAN 
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Good practices... 


Frame all questions as 
positive statements even if 
you are looking for 
problems 


Use the same scale on all 
questions 


Choose the number of cells 
on the scale, that Is, five or 
seven, so that you will get 
at least five responses in 
each cell 
More cells are better, 
subject to the constraint of 
five ina cell 

NORTHROP GRUMMAN 
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Several types of Likert scales exist... 


Likert scales exist for... 
= Quality 
= Importance 
= Agreement 
= Frequency 
= Likelinood 


NORTHROP GRUMMAN 
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Scales for agreement... 


Strongly Disagree Somewhat Undecided Somewhat Agree Strongly 
Disagree Disagree Agee Agree 


Scales need 
equal opportunity 
for positive and 
negative 
responses 


Strongly 
Agee 


Strongly 
Disagree 


Disagree || Undecided 


NORTHROP GRUMMAN 
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Examples of scales for frequency... 


About Half 
About Half |} Occasion- 
the Time ally 


NORTHROP GRUMMAN 
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Examples of scales for likelihood... 


Not At All 
True 


Probably 
Not 


a aes Probably || Frequently || Definitely 


NORTHROP GRUMMAN 
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Example of scales for quality... 


Extremely Below Above 
Average Excellent 
Poor Average Average 


Very Poor 


oS Very Good 


NORTHROP GRUMMAN 
36 


Copyright 2006 Northrop Grumman Corporation 


Example for importance... 


Moder- 
ately 
Important 


: Of Little 
Unimport- 
Import- 
ant 
ance 


Very 
Important Important 


NORTHROP GRUMMAN 
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Sample survey... 


Customer Satisfaction Survey Exit this survey >> 6. Northrop Grumman delivers on their commitments. 
Never Seldom pace tak the Usually Always 
1. Northrop Grumman provides systems that are easy to use. 
2 Frequency: J J J 7) J 
Strongly F ’ Strongly 
Disagree Disagree Undecided Agree Agree 
iS sag “ = al ~ ad 7. Northrop Grumman provides us with systems that work as they should. 
Never Seldom seat Har me Usually Always 
2. Allin all, I am satisfied with the products and services that I receive 
Frequency: J J J J J 


from Northrop Grumman. 


Seon: Disagree Undecided Serongly 
Disagree Agree 
3 . Northrop Grumman provides us good value with their systems and 
Agreement: 3 
g ~ ~ 2 J ae 


About 
. ase a al : Baty Halfway Mostly True True 
3. Northrop Grumman makes commitments that meet my n True Slightly True True 
About Half the We 
Never Seldom Time Usually kelihood: y a J J J 
Frequency: J J J J 


orthrop Grumman treats me like they want my business. 
About Half the 


4. When my system goes down, Northrop Grumman restores it quic Never Seldom Time Lstanally Always 
About Half the 
Never Seldom Time Usually J J J J J 
Frequency: J L» J J 


hen I need help, Northrop Grumman responds quickly. 


Not At All Only About Half 


True Slightly True True Mostly True we 


5. Northrop Grumman provides systems with the functionalit 


need. 
Never Seldom bait oy ie = - - = = 
Time 
Frequency: J a, J Done >> 


* Survey created on 
SurveyMonkey.COM 
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NORTHROP GRUMMAN 


Sample Results... 
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Analyze with ordinal logistic regression... 


= Conventional least 
Squares regression 


Response Information 


Variable Value Count 


ee : ; doesn't work on this kind 
: é of data 
c 9 ; 
Total 25 = Question #2 is set as the 
response variable, t.e., the 
Logistic Regression Table y-variable 
Predictor Coef SE Coeft z E = All others are set as 
Const{l} 25.2405 14.2487 1.77 0.076 : 
Const {2} 31.5295 16.0354 1.97 o.049 predictor variables, i.e., 
Const{3} Al.A778 20.1191 2.06 0.039 
a3 -3.96040 3.95140 -2.12 oO.034 the x-variables 
4 -0,841817 0.847503 -0.99 0.321 
a5 0.296463 0.775667 0.38 o.702 M Question #3 shows a 
a6 0.127389 oO.742109 O.17 O.864 a ; 
a7 0.401630 0.750945 0.53 0.593 significant relationship as 
a -1.35140 1.35373 -1.00 0.318 its p-value is less than 
* Regression Is performed with .Q5; all others are “noise. 
Minitab® Release 14.2 NORTHROP GRUMMAN 
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Throwing out all the “noise” leaves a 
prediction equation based on Q3... 


Response Information 


Variable Value Count 


Oe 2 1 
3 zZ 
4 13 
5 q 
Total 25 


Logistic Regression Table 


Odds 95% CI 
Predictor Coef SE Coet rs P Ratio Lower Upper 
Const(li 13.5565 4.46666 3.03 0.002 
Const(2) 16.6461 5.32721 3.50 0.000 
Const(3) 25,4195 6.58253 3.86 0.000 
O3 -5.60256 1.45034 -3.66 0.000 Oo.00 0.00 O.06 


Log-Likélihood = -7.642 
Test that all slopes are Zero: G = 36.646, DF = 1, FP-Yalue = 0.000 


NORTHROP GRUMMAN 
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Q3 proves to be a good predictor 
of the response to Q2... 


Probability —>. 
J that Q2 is 
—W, 90% 1,2,3, or 4 


Probability that Probability 


70% Q2 is 1,2, or 3 that Q2 is 
1,2,3,4, or 5 


Cumulative Probability 


1 2 3 4 5 


P(Q2) = © -5.60« Q3+ 13.56 Question 3 Response 
1+ e@ 5.60 x Q3 + 13.56 NORTHROP GRUMMAN 
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Choose improvements with a Pugh 
concept selection matrix... 


Better * 1.0 
Same 0.0 
Worse ™~ -1.0 


Direction of Improvement 


Direction of Improvement 


Proposal #1 
Proposal #2 
Proposal #3 
Proposal #4 


Importance 
Status Quo 


Maximize t 1.0 
Target * 0.0 
Minimize + -1.0 


1 


Sum of Correlation Values 
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Best choice 


* Pugh Matrix is produced with 
QFD/Capture Professional Edition 
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= Since Customers are the 


only ones that define 
quality, it’s useful to get 
their “Voice” in selecting 
improvement projects 


Statistic methods like 
logistic regression can be 
used to infer Customer's 
importance from survey 
responses 


Combine VoC with other 
factors in a Pugh matrix to 
aide selections 


NORTHROP GRUMMAN 
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Using Business Process Management 
Technology to Implement a CMMI-compliant 
Agile Software Development Approach 


Bob Moore, Business Transformation Institute, Inc. 


| , Business 


; Transformation Intro duction 
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¢ This presentation will show how Business 
Process Management (BPM) technology may be 
used to create process support tools that let 
Scrum developers be agile while still being 
CMMI-compliant. 
- Scrum is a popular agile software development 


method developed by Ken Schwaber and Jeff 
Sutherland in the early 1990s. 


fig What is BPM? 
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¢ BPM tools are next-generation software development 
tools oriented to process implementation. 


¢ The BPM community has blossomed in the past five 
years: 
- There are more than 50 leading tools. 
- Gartner Group predicts the BPM tool license market alone (not 
including implementation services) will exceed $3B in 2006 . 
¢ BPM provides a rapid, low-cost means to develop 
process support tools that are designed to meet each 
organization's specific processes. 


_) Frostman BPM Capabilities 
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¢ BPM is more than work-flow automation, though work- 
flow and BPM tools have much in common. 


¢ Th 


e typical capabilities of BPM tools are to: 
Model a process, typically in a graphical format, 


Integrate a variety of processes, external applications, and 
databases with the defined process, 


Manage step-by-step process execution across multiple 
personnel roles, 


Create exception handling and alternative processes, 
Monitor the health and fulfillment cycle of the process, 


Simulate the execution of the defined process based on either 
empirical results or user-provided parameters, 


Assign roles to personnel either by user direction within the 
process or based on current workload queues, and 


Collect metrics on process execution. 
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¢ This presentation is based on the use of 
Metastorm Inc.’s “Metatstorm BPM” tool 
(formerly e-Work) because: 


- It is prominent in Gartner's survey of BPM tools on a 
continuing year-to-year basis, 

- It has an appreciable share of the BPM license market, 
and 

- The author’s familiarity with the use of the tool. 
(Most important!) 


, Business 


— Vtransirmation What are Agile Software Development 


-_ Methods? 


‘ “The Manifesto for Agile Software Development” 
summarizes agile methods as those that value: 


“Individuals and over processes and tools 

interactions 

Working software over comprehensive 
documentation 

Customer collaboration over contract negotiation 

Responding to change over following a plan 


That is, while there is value in the items on 
the right, we value the items on the left more.” 
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, Business 


__tenomaion — SQme Scrum Vocabulary 


The project leader is called a Scrum Master — abbreviated 
here as SM. 


The list of requirements to be implemented is called a 


product backlog. 


One cycle in the project life cycle is a sprint. Sprints last 
for 30 days. 


The list of requirements to be worked in a particular 
sprint is a sprint backlog. 


A daily status meeting is a daily scrum. 


That accomplishments of the sprint are reviewed and 
demonstrated to the customer at the end of each sprint. 


Lo iresormation What is Scrum? 


" tute 


e@2e 
ros ~ 
| {Daily Scrum © 
According to Ken § (Every 24 Mioeme, 
Schwaber’s “, Hours) ‘ 
® of 


Agile Project 
Management 


with Scrum 
Scrum operates 
as follows: 


New functionality 
is demonstrated 
at end of Sprint 


Selected 
Product 
Backlog 


Project team 
and customers 


Product Backlog: 
Emerging, prioritized 
requirements 


Vision: 
Anticipated ROI, 2 
Releases, Milestones 
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. ; Transformation 


-_ The Agile Development vs. CMMI Challenge 


¢ Agile software developers often believe that 
CMMI compliance and the benefits of an agile 
approach are incompatible. 

¢ Likewise, many CMMI practitioners view the 
typical agile implementation as a “do what you 
want to do” rather than a disciplined 
environment. 


¢ These perceptions are false, yet they persist. 


Business 


___Irnsomation ~ Why Do Agile Developers Feel Agile is 
7 Incompatible with CMMI? 


. y 


¢ Many agile “gurus” characterize CMMI 
implementations as documentation-heavy and 
more interested in process adherence than 
satisfying customer needs. 


¢ Agile thinking is often somewhat counter- 
cultural and CMMI is viewed as a creation of the 
military-industrial establishment. 
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__Irmsjormation VWhy Do CMMI Implementers View Agile 
i Methods Like Scrum as Undisciplined? 


¢ Scrum is an empirical process control method. This 
tends to emphasize: 


- Short planning cycles— hence fewer formally documented 
planning artifacts that are used over long periods. 

- Personal interaction among the development team — hence fewer 
formally documented project monitoring and control artifacts. 

- Personal interaction between the team and the customer — hence 
fewer formally documented requirements management artifacts. 

- Adjusting the process implementation to suit the situation as it is 
actually being experienced — hence PPQA process compliance 
audits are less clear. 
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1. CMMI and Scrum are compatible. 

- This statement is offered without direct proof. 

- However, compatibility will be demonstrated using examples 


from a BPM implementation of Scrum processes with mapping 
to CMMI artifact requirements. 


2. Being able to achieve Maturity Level 2 or better is 
sufficient evidence of a process’s compliance with 
CMMI. 


- Note: Scrum contains a subset of the ML 2 specific and generic 
goals: 
¢ Scrum addresses many practices associated with PP, PMC, and 

REQM. 
¢ The implementing project or organization must still address CM, 
PPQA, and SAM in addition to Scrum implementation. 


¢ Some aspects of MA are covered by implementation of the Scrum 
processes through the BPM tool. 
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L Vinson FLow Does the BPM Implementation of 
= , Scrum Support CMMI? 
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¢ BPM supports Scrum in a CMMI context in two ways: 

- By supporting the natural implementation of Scrum in a manner 
that does not differ from the way developers expect Scrum to be 
implemented: 

¢ The BPM tool provides sprint planning and management tools. 
- By accumulating direct and indirect artifacts needed for a 
successful CMMI appraisal: 
¢ Capturing planning, project monitoring, requirements 
management, and stakeholder interaction artifacts without 
interfering in the Scrum implementation, 


¢ Capturing key metrics associated with the processes, and 


¢ Providing process audit trails that support CMMI appraisals and 
PPOQA audits. 
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' Business e 
__ Transformation How We Will Proceed 


é = 
oe 


‘eA 


¢ The BPM environment can support process 
implementation much more comprehensively than 
described here. 


— However, in this case the BPM environment was used to 
implement a Scrum process, so support is limited to what Scrum 
itself does. 


- For more information on general BPM support of CMMI 
implementation, see http:/ /www.biztransform.net/ 
¢ We won't cover every aspect of every process, but 
provide the highlights to guide the way to how CMMI 
and Scrum are mutually reinforced through the BPM 
tool support. 
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Business 


Transformation "Tactical Working Agreement 
Process Map 


¢ This is a high-level agreement with the customer 
that covers the general work objectives. 


- Not standard Scrum terminology. 
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, Business 


: Transformation Specific Practices 
(a Institute : : 
‘ = General Discussion 


‘eA 


¢ At the simplest level of implementation, a BPM tool 
provides a static electronic checklist that requires the 
user to record the actions that he or she is executing 
while following the process. 
- The checklist, like any quality checklist, is built to reflect the 
steps and roles involved in the process. 


- As the checklist is completed inside the BPM environment, the 
tool records an indelible audit trail of who completed each 
checklist step and when the step was completed. 

¢ This audit trail can be used to monitor implementation of the 
process. 


¢ The audit trail is also an indirect artifact that can be used ina CMMI 
appraisal. 
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_ Business 
. : Transformation 


LED stivut Specific Practices 
= General Discussion (continued) 


iy 


¢ Inamore sophisticated implementation, the BPM 
Scrum tool provides an executable process 
checklist for PP, PMC, and REQM practices. 


- The tool begins by allowing selected users to 
commence a process. 


For example, start prioritizing the product backlog with the 
customer (REQM). 


- As the process begins, the user is presented with 
potential next steps that can be taken in the process 


For example, determine customer sub-objectives for the next 


sprint given the customer's stated objectives (needs and wants) 
(REQM). 
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Geer Specific Practices 


—<= Institute 2 : ° 
= General Discussion (continued) 


- At each step along the way, the tool informs each user 
involved in the process about the list of process steps 
that the user must execute due to a triggering activity. 

¢ For example, in response to the team members completing 
their estimates of how much time a sub-objective task will 
take (PP), the tool presents the SM with the team’s estimates 
and prompting the SM to finalize the sprint plan (PP). 

—~ The BPM tool also records an audit trail of activities 
completed, completion dates and times, and 
personnel involved. It also prepares end-of-sprint 
reports used by management to understand progress 
and status (PMC). 


alg) 


Business 


__Trasformation Specific Practices 
Sia General Discussion (continued) 


¢ From the CMMI appraisal perspective, the tool 
captures: 


- Direct artifacts: outputs from executing the process 
(e.g., product backlog discussions and task estimates) 
with respect to: 

¢ PP—Specific practices 1.1, 1.2, 1.4, 2.1, 2.2) 2g 
¢ PMC—Specific practices 1.1, 1.2, 1.3, 1.5, 2.1, 2.2, 2.3 
¢ REQM —Specific practices 1.1, 1.2, 1.3 

— Indirect artifacts: the audit trail is an indirect artifact 

for the SPs listed above. 
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|) transformation Measurement and Analysis 
Ssaaoe= Institute ef ° 
| = Specific Practices 2.1-2.4 


- 


¢ The BPM Scrum tool provides the means to collect data 
on any given task’s execution duration. 


¢ This data can be stored and run through a variety of 
different analyses. 


¢ The raw data, and the analyses, can be automatically 
exported to other tools or sent directly to a 
“management dashboard” that displays measurement 
information. 


¢ From an appraisal perspective, the tool provides: 

— Direct artifacts: collected data and records of data collection, 
records of analyses, evidence of data and analyses storage, 
evidence of data reporting to managers 

- Indirect artifact: the tool itself is a resource for MA and can be 
used as an indirect artifact. 
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| , Business 


_) Frmsormaton Generic Practice 2.1 
= “7 (PP, REQM, PMC) 


, 7 


¢ Although the BPM Scrum tool can help in 
advertising a policy’s existence by reminding 
users of the policy at crucial process steps, it 
cannot create a policy. 


¢ As we have implemented it, the Scrum process 
as represented inside the BPM environment is 
the only way to accomplish the activities 
associated with certain PP-, PMC-, and REOQM- 
related processes. 
- For example, the only way to get review and 
approval of a sprint plan is through the sub-objective 


construction and task estimation process in the BPM 
tool. 22 


, Business 


; Transformation Generic Practice 2.1 


Agar stitute 
| = (continued) 


Sy 


¢ Given that a policy’s essence is that it expresses the 
expectations of an organization’s leaders, this “locking 
down” of how work may be performed is an expression 
of the policy. 

- If management declares that the product backlog prioritization 
and task estimation processes must be followed, and no other 
means of accomplishing a desired result is available except 
through the BPM Scrum tool then adherence to management's 
policy is more effective than any written policy document could 


ever be. 
¢ From the appraisal perspective, the tool provides: 


- Direct artifacts: Demonstration using the BPM Scrum tool of the 
enforcement of the processes as described above in the specific 
practice discussions for the PP, PMC, and REOM process areas. 


- Affirmations: Testimony that there is no alternative method for 
accomplishing the intended results of the process. 
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| transformation Generic Practice 2.2 
= “7 (PP, REQM, PMC) 


, 


¢ Planning a process involves laying out the 
approach to be taken, in the context of an 
organization or project, to implement the 
process. 


¢ The BPM-implemented Scrum process does not 
provide all the plans to be used by the project in 
implementing all process. 

—- The BPM Scrum tool does put into practice the 
process to be followed in creating particular plans, 
such as software development and project 
management plans. 
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| | Institute 
= (PP, REQM, PMC) [continued] 


, Business 


Transformation Generic Practice D ° 2 


, UY 


¢ From the appraisal perspective, the tool 
provides: 


- Direct artifacts: The BPM tool itself encapsulates the 
plan for some PP-, PMC-, and REQM-related 
processes since it creates and publishes the Working 
Agreement (contract equivalent), Service Delivery 
Plan (project plan equivalent and requirements 
specifications), and the 30 day sprint plan 
(requirements analysis and task planning). 


- Indirect artifacts: Records of the activities completed, 
completion dates, and personnel involved in 
executing the planning processes. 


AS 


| ETransjormatin Generic Practice 2.3 
2 (PP, REQM) 


t 7 


¢ The BPM Scrum tool has a directory of 
personnel in various roles that may perform 
work in implementing the software project. 


¢ At the start of each overall project and sprint 
planning cycle, the BPM Scrum tool provides 
this directory to the SM as a list of names to be 
assigned to working on the customer's sub- 
objectives (requirements), working agreement 
creation, and service delivery plan creation. 


¢ The tool causes each personnel resource to 
estimate their availability on a weekly basis 


throughout the duration of the next sprint. 
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, Business 


. ena Generic Practice 2.3 


| = a (PP, REQM) [continued] 


, 


¢ From the appraisal perspective, the tool 
provides: 


— Direct artifacts: 


¢ The BPM Scrum tool is a resource in supporting the planning 
process, so the existence of the process as implemented in the 
BPM tool is itself an artifact for GP 2.3 in PP and REOM. 


¢ The records of personnel availability and subsequent 
assignment to work based on task estimates are also direct 
artifacts. 

- Indirect artifacts: the audit trail records showing the 
assigned personnel working on the requirements, 
project planning, and project implementation 
processes. 


Ai 


Br sformaton Generic Practice 2.4 


i (PP, PMC, REQM) 


‘es 


¢ The BPM Scrum tool helps aSM assign 
responsibility in two ways: 

- Each Scrum process has a set of roles associated with 
each process step. This set of roles provides the basis 
for assigning responsibility. 

- The tool provides the SM with an inventory of 
personnel who are available to work on a process, to 
whom the SM can then assign responsibilities. 


¢ As SM associates personnel with the process, the tool 
documents this assignment and informs the personnel thus 
assigned about their responsibilities as the process is 
executed. 
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| , Business 


| Prasat Generic Practice 2.4 
= ~~ (PP, PMC, REQM) [continued] 


- 


¢ From an appraisal perspective, the tool provides: 


- Direct artifacts: records of assignment of personnel to 
roles and the association of roles with Scrum 
activities. 


- Indirect artifacts: The record of personnel fulfilling 


their assigned responsibilities as captured by the BPM 
Scrum tool's audit trail. 


Zo 


Br sformaton Generic Practice 2.5 
Ssaow> Institute 
| = (all supported Process Areas) 


, 


¢ The BPM Scrum tool provides context-sensitive 
help for users engaged in any given process 
step. 

¢ The tool also guides the user through 
implementing the process by limiting their 
choices with respect to the next possible steps. 

¢ From an appraisal perspective, the tool: 


- Direct artifacts: records of the user referencing the 
context-sensitive help (a weak direct) 


- Indirect artifacts: none 
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Bi cnsjormaion Generic Practice 2.6 


-_ (PP, PMC, REQM) 


‘eA 


¢ A BPM Scrum tool does not directly manage the 
configuration of items produced by the process. 


¢ However, the tool can forward designated products 
resulting from process execution to the configuration 
management (CM) system. 

- For example, when a sprint plan is being circulated during 
construction and approval, the BPM tool can capture the plan at 
various stages and provide it to the CM system for version 
control or baselining. 

¢ The BPM Scrum tool is part of the CM system since the 
products to be controlled through this mechanism may 
be selected from the PP-, PMC-, and REOM-related PA’s 
designated configuration (or control) items (CIs) and the 
stages at which the Cls are to be captured is 
programmed into the BPM Scrum tool. 31 


| Transformation Generic Practice 2.6 
= ne (PP, PMC, REQM) [continued] 


=, 


¢ From an appraisal perspective, the tool provides: 


- Direct artifacts: a demonstration of the BPM tool’s 
ability to capture Cls at different stages in support of 
the CM system is a direct artifact. 


-— Indirect artifacts: none 
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eee Generic Practice 2.7 


-_ (PP, PMC, REQM) 


‘eA 


¢ By its nature, a BPM-implemented process requires the 
identification and involvement of stakeholders who are 
relevant to each process step. 


- As any given process (e.g., processes related to PP, PMC, REQM) 
is programmed in the BPM environment, the roles associated 
with each process step must be designated. 

- Either before or during process execution, these roles are 
associated with specific personnel. 

- These personnel are the CMMI's “relevant stakeholders”. 

- As the process executes in the BPM Scrum environment, these 
personnel are involved in whichever step(s) of the process is 
assigned to them. 
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/ LTrasjrmation Generic Practice 2.7 
= ~~ (PP, PMC, REQM) [continued] 


=, 


¢ From an appraisal perspective, the tool provides: 


— Direct artifacts: 


¢ The creation of process roles and the assignment of 
personnel to these roles are jointly demonstrate “identifying 
stakeholders”. T 

¢ The BPM Scrum tool audit trail demonstrates “involving 
relevant stakeholders” requirement by recording the 
involvement of the designated personnel in the processes. 


- Indirect artifacts: none. 
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' Business 


. 7 Wena’ Generic Practice 2.8 
= ~ (PP, PMC, REQM, technical activities) 


¢ The BPM Scrum tool assists the SM in monitoring and 
controlling a process in two ways: 

- The SM can monitor project participants recording of their daily 
Scrum meeting minutes, recording of time spent on tasks, and 
completion of assigned sprint tasks in real time through his or 
her process “watch list”. 

- The BPM Scrum environment can also feed data into a 
“management dashboard” that provides metrics on the sprint’s 
status with respect to tasks completed, amount of time spent on 
tasks, etc. 


¢ From an appraisal perspective, the tool provides 


- Direct artifact: the tools ability to provide information, along 
with proof that management is using these functions, jointly 
provide a direct artifact. 


- Indirect artifact: none. 


315) 


ee Generic Practice 2.9 


-_ (all supported Process Areas) 


¢ The BPM Scrum tool does not itself directly provide 
objective insight into the adherence of PP-, C-, or 
REQM-related processes to their descriptions, standards, 
Cle: 


¢ However, BPM Scrum helps an organization comply 
with generic practice 2.9 as follows: 


- First, implementation of a process (e.g., sprint reviews, sprint 
planning, backlog prioritization) in the tool means that the 
personnel who are executing the process cannot deviate from the 
process as defined. 

- The inability to circumvent process steps (or the entire process) 
reduces the need to monitor process execution as closely as is 
normally required with “paper-based” processes. 

- Additionally, the BPM Scrum tool process audit trail provides a 
log of the process steps performed, by whom the steps were 
performed, and when they were performed. 
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' Business 


_) Fresormaton Generic Practice 2.9 
| = (all supported Process Areas) 
| [continued] 


- 9 


¢ From an appraisal perspective, the tool provides: 
- Direct artifacts: none. 


- Indirect artifacts: the audit trail is an indirect artifact 
if it is used in assessing process compliance. 


iil 


EE Generic Practice 2.10 


Institute 


| (all supported Process Areas) 


‘Fs 


¢ The BPM Scrum tool supports higher-level management's reviews 
with respect to a process in a several ways: 

- The higher level manager can have a set of process activities (e.g., sprint 
planning, task status) on his or her “watch list”. This enables the 
manger to view activity status of the processes at any time. 

- The BPM Scrum tool can also feed data into a “management 
dashboard” that provides measurement data on sprint planning 
duration, task duration, requirements approved by the customer. 

¢ From an appraisal perspective, the tool provides: 

- Direct artifacts: a demonstration of the BPM tool’s ability to provide 
information, along with proof that this information is supporting higher 
level management's reviews, is jointly a direct artifact. 


- Indirect artifacts: none. 
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: Transformation Su MimMary 


é = 
| | | 


‘eA 


¢ BPM supports Scrum in conjunction with CMMI by: 
- Helping the Scrum Master and project team members plan and 
manage sprints. 


- By accumulating direct and indirect artifacts needed for a 
successful CMMI appraisal without project members realizing 
the CMMI is going on in the background. 


¢ This presentation covers the basics. 
- For more information about applying BPM in a CMMI context, 


see http://www.biztransform.net/ 


- For more information on Scrum see any of the books published 
by Ken Schwaber (just ignore what he says about CMM/CMMI 
processes). 
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¢ Bob Moore, Business Transformation Institute, Inc. 


- rlmoore@biztransform.net 
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Sender Receiver — 


® Unfiltered communication 

& Environmental factors 

Ee Input Selection challenges 
— Mission and objectives clarity 


— Sender coordination 
— Recipient filtering 


Potential Issue — Noise via Distortion 


Sender Receiver 


Context mismatch 
Medium unreliability 
Faulty encoding/decoding 
Synchronization issues 
Mixed Messages 
Interrupted signals 
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® Typical Errors 
Confusing the model with process 
Neglecting the bottom line 
Compartmentalization of improvement efforts 
e By project domain 
e By function 
— Maturity Level Mandates 
& Common consequence: inability to measure and 
discuss impact of changes 


& Usual Outcome: using the documentation 
(medium) as indicators of Success 


How Issues Arise 


© 2004 Apogen Technologies 


® Encoding, decoding, context issues often are 
intangible 

& The medium is tangible and attracts attention 

® The “medium over the message” syndrome 
surfaces 

® Focusing on the medium can aggravate noise 
issues, resulting in a vicious cycle 


Common Symptoms of Media Fixation 


m» Format Wars 


& Referring to documents (not the combinations of 
tools, personnel, and methods) as “the process.” 


& Top-down-only view of process definition 
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& The model implies all components of 
communication loops 

& The model doesn’t tell us specifically what we 
should say 

® The model does indicate what we should be able 
to communicate about 
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= Communication Infrastructure 
— Sustaining channels (media) 
— Providing access 


& Communication Relationships 
— Management-project communications 
— External communications 
— Intra-group communications within projects 
— Communications between projects and supporting groups 


& Communication Contexts 
Product development 

Process management activities 
Commitments and agreements 
Action item management 
Status reporting 


Odd iulerslmarslelicas 
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= Common lexicon and operational definitions 
® Understanding of business context at applicable 


levels 


— The model contains roughly 75 references to business 
context, drivers, situations, etc. 


e Identification of appropriate stakeholders 


Examples from the Specific Practices 
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® Product Development 
— RD.SP.3.2 “Establish a definition of required functionality” > 
*...communicates the manner in which the product will be used.” 
— PI.GP.2.7 “Communicating new, effective product integration 
processes to give affected people the opportunity to improve their 
performance” 


® Process Management 
— RD.SP.3.2 “Establish a definition of required functionality” > 
“...communicates the manner in which the product will be used.” 


— PI.GP.2.7 “Communicating new, effective product integration 
processes to give affected people the opportunity to improve their 
performance” 

— MA.SP.1.2.3 “Specify Operational Definitions” > “Communications” 
establishes context for communication 


TECHNOLOGIES 
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® Agreements 


— SAM.SP.1.3 “Establish Supplier Agreements” implies a host of 
communications. 


e Action Item Management 


— VER.SP.2.2 “When issues arise during the peer review, they should 
be communicated to the primary developer of the work product for 
correction.” 


— PPQA.SP.2.1 “Communicate and ensure resolution of 
noncompliance issues” 


® Status Reporting 


— PMC.SP.1.3.3 “Monitor project risks” > “Communicate risk status to 
relevant stakeholders.” 

— PMC.SP.1.6.1 “Conduct Progress Reviews” > “Regularly communicate 
status on assigned activities and work products to relevant 
stakeholders “ 

— CM.SP.3.1.2 “Ensure...access,” Examples of “...communicating 
configuration status.” 

— MA.SP.2.4 “Communicate Results” AP F 

OG 
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Summary 
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& The CMMI describes communications, both 
explicitly and by implication 

& Media is a critical element, but not the only one 

= The media serves the messages in communication 


® Improper focus on the media may distort or 
completely subvert the message 
® Understanding complete communication loops and 


how the CMMI relates to them preserves the critical 
messages 
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Core Capabilities 


e Seekers and electro-optical e Materials and processes 
payloads composites, optical elements 
and coating metallurgy 


¢ MMIC 


e Guidance and navigation 


e Image processing - trackers 
oo ¢ Antennas and radomes 
e Rocket motors 
e Electronic warfare 


e Explosives 
e Communication, secure data 
e Warheads and fuses links 
* Acoustics * Terminal ballistics 
* Thermal batteries ¢ Simulators, trainers, training 


; systems 
e High power microwave y 


e Armor and protection 
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Excellence through Standards 


2006 ISO 14001:2004 
CMMI-3 
2005 AS 9100 
RAFAEL LTD. ISO 90003 
HAS BEEN AWARDED i$a™. ISO 9001:2000 
The 2003 = 2003-4 
National Quality Award é Sete OHSAS 18001 
in Industry ‘a 
— ISO 17025 
2002 { 
ISO 14001 
2001 CMM-3 
ISO GUIDE 25 
1996 { 
ISO 9000-3 
1993 ISO 9001 
1989 Mil-Q-9858A 
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e Essence of Maturity Level 3 


CL3 Project Ea 
Management Deployment 
Quantitative CL3 Organizational 
Project Engineering Process 


Management Performance 
Integrated Validation vy. 
Supplier fz Organizational iedeceenoet 
Management__ enaline } Organizational 
Integrated > oe a Environment for 
Teaming Product iy Ongena Integration 
Integration , 


Process 
Risk 


| Causal 
Analysis and 
Resolution 


Definition Decision 


Technical y 
| Management | ee : 
Solution | Organizational Analysis and 
Project oe ae Focus 
Management Yessy) ates 


Maturity Levels (Staged Approach) 


Project Management Process Management 
Categories (continuous Approach) 
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The Essence of Capability Level 3 


e Generic Goal 3 


— The process Is institutionalized as a 
defined process 


e Generic Practice 3.1 
— Establish and maintain the description of a defined 
process 
e Generic Practice 3.2 
— Collect work products, measures, measurement 
results, and improvement information derived from 
planning and performing the process to support the 
future use and improvement of the organization’s 
processes and process assets 
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The Organization at Maturity Level 3 


Information 
1 PF 2 . 
pEeee ace See sta 
Improvement Processes 
Improvement Improvement 
Opportunities Activities Organizational 
Training 
OPD/SG1 
GP 3.1+IPM 
= ei : Organizational Defined 
pee Process Process 
Process 


Assets 


Engineering GP 3.2 Project Management 


Information ; 
Proj. 
for Process ° 
Improvement 
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Same Information — Different Use Contexts! 
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EGAN DAN ofl Gs Corporate Management 
Organizational Process Assets 


e The organization's process asset library is a 
collection of items maintained by the organization for 
use by the people and projects of the organization 

e This collection of items includes: 

— descriptions of processes and process elements 
— descriptions of life-cycle models 
— process tailoring guidelines 

process-related documentation 

process-related data 

e The organization’s process asset library supports 

organizational learning and process improvement by 

allowing the sharing of best practices and lessons 
learned across the organization. 
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Knowledge Management — The Motivation 


¢ At the Individual Level 
— Capability to perform tasks and to make decisions in a higher level 
of quality 
e« Process knowledge 
¢ Professional knowledge 
e« Business knowledge 
e Expert identification 
e At the Organization Level 
— Knowledge is the most important resource of the organization 
— Appropriate management of the knowledge improves organizational 
performance 
¢ Knowledge sharing and knowledge assets 
¢ Visible and invisible knowledge 
¢ Technology 
e Organizational processes and culture 
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RAFAEL’s KM Objectives 


e Support the Organizational Core Processes 
— Improve the effectiveness and efficiency of sales 
Improve project process efficiency 
e Reduce errors 
¢ Decrease “wheel inventions” 
Improve proficiency and effectiveness in development 
¢ Generate relevant connections among people 
¢ Improve the availability of relevant information and knowledge 
Improve field-test success ration 
Improve business intelligence in marketing and sales phases 
— Support of innovation in all areas 
e Implement Effective and Efficient Solutions 
— Construct appropriate KM infrastructure over all aspects 
¢ Technology 
¢ Process 
¢ Culture 
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Types of Needs 


e Knowledge Sharing by People with 
Common Subjects 

¢ Accessibility to Information within 
Specific Context 

¢ Handling “Know-How” 

¢ Accessibility to Professionals 

e¢ Decision Support 
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Types of Solutions 


e “Know-How” Management 


e After-Action Reviews and Lessons 
Learned 


¢ Knowledge Communities 
¢« Contents Management 
¢ Organizational Infrastructure 
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Knowledge Communities 
Definition 

— A group of people with a common subject 


¢ Not part of the organizational structure 


¢ Conduct dynamics of sharing 
— Meetings 
— Web site / Forum 


¢ Knowledge Community Objectives 
— Knowledge sharing within the group 
— Common learning from outer sources 
— Generating new knowledge 
— Organizational lobbying for the subject 


NDIA-CMMI Denver, CO, Nov. 06 RAFAEL’s Proprietary KM for CMMI-3 at RAFAEL 19 


Unclassitied 


Corporate Management 


Contents Management 


¢ Definition 
— Organization of contents in a form which enables convenient 
retrieval when needed 
¢ Contents Management Principles 
— Every contents item 
e Exists in exactly one location 
¢ Accessible from a large variety of locations 
¢ Access is according to relevant context 
¢ Contents Management Solutions 
— A large variety of solutions exist 
— A specific solution should be chosen after careful analysis of 
the needs 
— All solutions should be supported by the organizational 
infrastructure 
— “Network Based Contents Management” is the advanced 
type of solutions 
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Examples of KM Systems in RAFAEL 


e “Nympha’” 
— The knowledge source for PM and SE Processes 
e Will be discussed next 
e After Action Review (AAR) System 


— Methodology-based system to support AAR and lessons learned from 
failures and successes 


— Lessons and recommendations are structured in retrievable formats 
— Root cause statistics caters for process improvements 
e Shemesh — Business Information Source 
— An integrative system for storing and retrieving relevant business 
knowledge 
¢ Competitive Products 
Visitors 
Exhibitions 
Meetings 
Customers 
Business trips 
Contracts 
e Conference Participation Optimization 
— Classified conference catalog with priorities 
— Coordination between conference participants 
— Registration and report process 
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“Nympha” — The PM and SE Knowledge System 


e The Material is Organized in “Knowledge Drawers” 
(Process Areas) 
— E.g. Requirements Management, V&V, ILS 
e Each drawer has a “Drawer Manager” 
— A domain expert 
— Responsible for reviewing new materials 
— Responsible for updating the contents 
e Each drawer contains two types of items 
— Internal items — specific contents that do not exist elsewhere 


— Links to external items which exist in other sources on the 
network 
¢ Procedures and Methods Libraries 
e After Action Review System 
e Document Management System 
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Nympha Drawer: Requirements 


& http: //2.0.34.80 /nimfa-site- dbp /Templates /drawer aspx id= 32 - Microsoft Internet Explorer _ 


| eBack ~ + + OA | Gseach Giravorites Meda J) Gy So! a 
| Address ®@) http: //2.0.34.80/nimfa-site Weare {drawer aspx?fid=32 


nvr | soar aw? | af GRY 108 [oon @ [onsoo we? lone (wen) ___] nv’9Op4y 


Nw MwA man <= Peo Nw mang 


ONIMA AYLI? ATTY NaWwAN man 'Ol Tan? Tyo MWOTH O'O2 wit nwo ning 
DYNWYN DYER NINA WOR? NIWOT Pwo nwo? von 7a n2'20? .DIT TAN O'nMs 
pna naons Wy ywown? 


ytion miwan nei7yoa Arya DAWN If Aan? 1oldIw OWINX 
t (12%) niao7oXA TTA O"yyaAN DwINANA n7TaA (36%) nwt nvav nara 
Dv 68 10%) Orit m7! NTA (10%) mMaqayoi mn Amy (11%) Gow ee ayn ya HT 


mo vo'rorl 


a5569 nena or02 t wal net ona @ 
Requirements Development (RD) ‘uy'n jpn 2 PT ces ments 


Requirements Management (RM) yw'n jpn 


THIN NWT Mi, ~ 15.00.12 70 
IEEE 1220 qy'n pan T | 
omple IEEE 1220 gan jan 


tent tlt elle ce 20 een Wn e3 Requirements Analysis 


O90) On 


C 
con 


6.1.9 qyto IEEE 1220 ow'n in 
mw on oom/noan 


See next slide 
Lr 


a CT) lorem 


NDIA-CMMI Denver, CO, Nov. 06 RAFAEL’s Proprietary KM for CMMI-3 at RAFAEL 23 


Unclassitied 


Corporate Management 


Requirements Drawer Process: Requirements Analysis 
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Establish Req. Table Work-Product: Req. Table 
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Thank you for listening 


Any Questions? 
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@How to Measure 


@What to Measure 
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@Why to Measure We are often questioned about the cost & return of 
CMMI-based process improvement? 


@ Sponsor - Why sponsor? What is 
the value to business? 

@ Project Manager — Why process? 
My customer only care about the 
schedule 

@ Engineer - Why measure? Does it 
measure my performance? 


@Why to Measure 


@ Building continuous commitment and 
sponsorship 
@ Understanding the value of CMMI to 


Motorola business 
Process Improvement is oriented by business 
and is expected to benefit business 


@ Benchmarking with other 
organizations 

© Set up the baseline for continuous 
improvement 


@Why to Measure 
@ How to Measure 
@ What to Measure 
@ What to Be Measured 
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Lead the Result 


@ Performance Result & 
Benefit 
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* Busimess : 


Set 
@How to Measure 
WEES 
Performance in YU 


. ’ Analyze what are the business goal of 

SEUSS LI Analyze what’s & Quality T Trend 
Benchmark and know what’s our Position 

’ Calibrate the analysis result and Seuip business goal 


Know our performance 


Performance: 
Result 


Measure 


Measurement System Overview 


Project Level Focus 


Goal Setting (SQAP) 
Monthly Project Review 
Post-mortem/Causal Analysis 


GSG Lessons Learned , POR 


Process Data 
Site Level Focus Repository Process Assets and 


EIQMEn Standards 


Q Data & SQAP Template 


Yearly Goal Setting & 
Monthly Tracking Mechanism 


Org Data Audit Mechanism | SW Metrics Spec & Data 


(Org Audit + SQE Audit + Collection Guideline 
Tool Validation) 


Yearly Baseline Mechanism baie Nt & 
(YBL/PCB) 
Process Model Usage Guideline 
Quality Trend Analysis & POR 
Mechanism 
Metrics Automation Tools 
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Define Common 
Measures of 


Performance 
Review/lnspection 
Testing 
Risk Management 
ee RM 
| PP/PMC 
L SWMetrics |. fs 

Specification |) ~~ View! 


Project Metrics 


Organizational 
Metrics 
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View2 
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ts Process 
Project Data Collection Guideline Braduel 


Metrics Analysis Template & Guideline 


QPM Guideline 


Supporting Artifacts 


View3 


asure How Measure 


Business Partner Environment 


Defect Tracking 


EPMS Proj Mgmt 


Center Goals/Baselines 


Manage By Data 


Project Record 


Center Metrics Database 


Ci 


Project Metrics Database 


Change Mgmt 


Asset Characterization 


Australia 
Dev Team 


PMBOOK 
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@Why to Measure 


@How to Measure 


@What to Be 
Measured 


@What Changes to 
Lead the Result 


dd aie aaitlarers 
Result & Benefit 
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Process Initiatives : All major initiatives will have 
ROI analysis in POR System 
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How Measure 


Organization Performance Tracking 
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Chart 4: In-Process Faults for 
Development & Test 
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Why Measure How Measure What Measure Result of Measure 


Process Performance Baseline 
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@Why to Measure 
@How to Measure 


@What to Be 
Measured 


@What Changes to 
Lead the Result 


yo claielaiitslalers 
Result & Benefit 
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Process Architecture and Integrated Project 


Process 


@ GSG Process Hierarchy Structure 
Adapted to accommodate process changes due to CMMI. 


@ Tailoring Flexibility 
Organized to assist the projects on process model 
selection, and provides guidance and flexibility to select 
and tailor the processes at a detailed sub-process level. 


@ Easy Access 
The process asset library can be accessed by several 
criteria, including CMMI process areas, phase, process 
templates, checklists, and other types of documents. 
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Process Category 


Preceding Assets 


Figure 2-1 Overview of Organizational Process Hierarchy 
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Quantitative Project Management Process 


Focus more on critical processes aligned with business objectives, 
particularly in the areas of requirements management and 
verification. 


Understand Scorecard 
Project Planning | 
COQ,COPQ, Productivity ,Defect Leakage . 
,Customer Satisfaction, On-Time Ide nti fy 


Delivery... 


Project Monitoring 
&Control 
Verification 


Estimation Improvement, Test Req ulrement 
Process Improvement,etc M ana g ement 


Analyze Process 
Performance 


Process Capability Baseline, 


Process Opportunity 


Define,the Measures > Measure Seriennaiice ae Maniter Pe rmance} Stabilize Proces: 
Control Process » yaaa Process Performance Measure the Benefit Result 
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Using of High Level Process Area 


Techniques &Tools in Digital Six Sigma 


Aches al Uh ene a Quantified Root Causes 
Avi a NL = 
My \/ a Indicators | 2 
- & Problem 
a 
cr — 
ll. 


Statement 
(effect) : 


1. Provide analysis techniques 
2. Provide stat|stical method 
to understand yariation 


1. Provide analysis techniques 
2. Provide statibtical method 
to understand variation 


SPC methodjand tool to 1. Quantitatively identify 
support baseline and model improvement opportunities 
establishment 2. Cost-benefit analysis 

3. Measure improvement effect 
by performante analysis 


Defects and 
other 
problems 


Project Data 
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Project Management & Engineering 
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Support Verification Process Improvement 


@ CAR and OID-based processes were used to improve 
verification process effectiveness without compromising quality in 
specific classes of cases. 


@ The formal process rules the Center previously used were less 


effective for smaller documents and project teams. 
Normal Size 
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Inspection Process Improvement 


@ Mini-reviews and mini-inspections were introduced; effort and 
cost were reduced without affecting defect density. The results were 
used to establish new baselines for planning and quantitative project 


Management. 
Formaltinspection Meeting Performance 
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: We ri aR) ‘ zr ry eat 5, 3 >| | liye webs 16 


Moving Range 


| l 
l 
y i i | | h 1 
iy . bt le «pl As i sill I i ag a i i er a 
OS eis TS Fv 2S 


lay a 


eT ee Pe Tapy a senshi "eel 
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Enterprise Project Management System 


@ Use Enterprises Project Management System (EPMS) as a 
common project management platform to manage global team 
@ Use Project Pamphlet as a local supply tool of EPMS for PM 
performance tracking 


Staffing Effort 
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Requirement Management Process Improvement 


RM Improvement is based on guidance in CMMI 


@ Requirement Volatility 
Late additions and changes to requirements are measured. 


@RM Process Performance and Management 
F The project level results then are rolled up to the organizational level for 


senior management insight 
FSet control limits for future use in quantitative process management of 


requirements volatility. 


@Use of RM Measure Result 
The measured and estimated impact of the requirements changes 


FLed to new project processes for replanning, estimation, and 
FA factual basis for negotiation with customers. 


Better Manage the Change | } Better Project Management Delight the Customer 
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Requirement Management Process Improvement 


2003 -— Before CMMI 


I-MR Chart of RV by YEAR 


Individual ¥alue 
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Moving Range 
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Observation 
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Improvement Action: 
1. RM process 

2. IAD template 

3. RM TRMX template 
4. RV data sheet 


2004/2005 — After CMMI 


Individual Value 


Moving Range 


T-MR Chart of RV-1 by Year-1 
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Requirement Volatility - Effort 


—e— RV Effort- Cumulative % 
RV Trigger 
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Decision Analysis and Resolution 


@ New practices based on the CMMI Decision Analysis and 
Resolution (DAR) process area were used to improve the Center’s 


managerial and engineering processes, particularly those that map to 
Technical Solution, Configuration Management, and OID process areas 


@ These DAR-based practices have been applied to improve 
decision making effectiveness at both the organization and project 


level. Decision = ROI Analysis 


Weighted-Average 
Project Name: »0x« 


Figure 3 
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Shifted its focus on causal analysis processes from the 
relatively narrow defect prevention perspective in the SW- 
CMM to the broader problem prevention focus in CMMI 
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@Why to Measure 


@How to Measure 


@What to Be 
Measured 


What Changes to 
Lead the Result 


@ Performance 
Result & Benefit 
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fea er taliries Reduction of Cost of Quality 


Result & Benefit 


eReceye) 
; 


2003 


Before Chil Transition After CMMI Transition 


COQ Reduction by one-third from pre-CMMI 
baseline due to improvement in verification, peer 
review and software test process. 


» 


7 ne Improvement in Fault Density 
Result & Benefit Faults Per Thousands Line Of Code 


ea & & 
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2004 2005 
Before CMMI After CMMI Transition 
Transition 


IPF Reduction by 40% due to improved Casual 
Analysis and Resolution (CAR) procedures by 
implementing real time RCA and by improved 


verification process 


» 


Improvement in Estimation Accuracies 


@ Performance 
Result & Benefit 


First Estimation Accuracy 
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Before CMMI After CMMI Transition 
Transition 


Improvement in Effort Estimation accuracy by 
31% and Schedule duration by 84% - due to 
improved Project Planning and tracking practices. 
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Requirement Phase Containment 


@Performance 
Result & Benefit 


401 = 


2003 


Before CMMI Transition After CiMl Transition 


Requirements PCE sustained at 90% 


in spite of enormous organizational growth and 
constant requirement churn in number of projects. 


» 


Design Phase Containment 
Performance 


Result & Benefit 


401 = 


2003 


Before CiiMl Transition After CMMI Transition 


Design PCE improved by 8% 


Improvements were due to improved Verification, 
Technical Solution and Organization Training process 
areas 


» 


Code Phase Containment 
@Performance 
Result & Benefit 


401 = 


2003 


Before CuiMl Transition After QuiMl Transition 


Code PCE improved by 12% 


Improvements were due to improved Verification, 
Technical Solution and Organization Training 
process areas 


» 


Customer Satisfaction 
@Performance 


Result & Benefit Overall Customer Satisfaction 


ao a & 


TCS 


J Consistently received high TCS scores (9/10) with a 
sustained 100% on-time delivery rate 


J Continued to exceed customers’ expectations to show the 
ability to deliver cost effective, high quality products on 
time by providing excellent technical solutions and project 
execution. 


» 


Business Benefit 


@Why to Measure 


@How to Measure F Customer Satisfaction 
Sustain to exceed customer expectation (+9/10) 


@What to Be F Business Growth 
Vikerekolel caxe, GSG Population 2225 (Y2000) - > 6840 (2006) 


ri arsit Changes to FEngagement Capability 
Lead the Result Model Level->System Level->Product Level 


@ Performance 
Result & Benefit 
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@Why to Measure 
@ How to Measure 


@What to Be 
Measured 


@What Changes to 
Lead the Result 


Performance 
Result & Benefit 
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Raytheon 


Customer Success Is Our Mission 


Do you understand why Level 3 
comes before Level 4 which comes 
before Level 5? 


National Defense Industrial Association 
6th Annual CMMI® Technology Conference 
and User Group 
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: Raytheon 
Introduction 


e This presentation describes key items that help in the 
understanding and communication of the relationships between 
CMMI® Levels 3, 4, and 5, including 


— how Quantitative Project Management (QPM) activities 
build on the Measurement and Analysis (MA) activities 

— how Organizational Process Performance (OPP) 
feeds QPM and QPM feeds OPP 

— how Organizational Innovation and Deployment (OID) 
activities build on the Organizational Process Focus (OPF) activities 

— how QPM is tied to both the OID and Causal Analysis and Resolution 
(CAR) activities 

— how the Raytheon Six Sigma™ process relates to the CAR process area 

— the Level 4 and 5 inter-relationships from the program and organization 
perspectives 


Raytheon Six Sigma is a trademark of Raytheon Company. 
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Importance of Understanding the = ®aytiheon 
Relationships 


e Consistency in CMMI interpretation is essential to a 
successful appraisal 


e The organization focus should be on “Does it make good 
business sense and does it meet the intent of the CMMI 
model?” 

e Understanding the Level 4 and 5 relationships within the 
CMMI model helps clarify how the business meets the intent 
of the model 
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Raytheon 
Process to gain understanding 


After Class C SCAMPIs it was clear there was: 


— Misunderstanding and miscommunication about the Level 4 and 5 
interpretation and expectations 


To address the issue, our organization held an in-depth 
model review of the CMMI Level 4 and 5 process areas 
with key members of both the 
appraisal team and the 
Enterprise Process Group 
(EPG) 

— Performed an in-depth model review 
— Documented our understanding 

— Mapped our processes to the model 
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Raytheon 
Results of the Meetings 


e The time spent in coming to a common understanding of the 
model helped: 


— Clarify the key interfaces amongst the 
Level 4 and 5 practices and how they 
build on a Level 3 organization 


— Reveal how our organization is 
meeting the intent of the practices 


— Increase the programs’ 
understanding of the information 
needed to provide evidence of 
their Level 4 and 5 activities. 
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Raytheon 
Discussions 


e The CMMI context diagrams were used as the starting point for 
discussion 


¢ Discussions included ttt aoa HP 3 
a | ee) peer 7 ores: | 
— The relationships of the esi ee (=) a 
specific practices within eras a Ee 
a process area Ser Nee See) 
— The interfaces between —— 
specific practices of each antiy “Record 
process area to other process bata 
areas 


— Types of evidence expected for 
both the specific and generic practices 


e The following slides address some of the key items that help in the 
understanding and communication of the relationships 
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Organizational Process Raytheon 
Performance (OPP) 


e The organization needs to be operating at Level 3 in order to 
perform the OPP activities 


¢ For Level 4, the organization adds 3 il be 


— Statistical analysis of its process capability 
based on historic data 
= Generation of process performance baselines 
= Development of process performance predictive models 


— Definition of the organization’s quantitative quality and process 
performance goals and objectives 


— Direction and support to the programs on the use of the organization's 
defined goals and objectives 
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4. Quantitative Project Management = aytheon 


(QPM) 
e Builds on Measurement and Organization 
Analysis foundation Measurement and Analysis 
¢ Includes quality and process- Program 
performance objectives (program, Measurement 
customer and organization) and Analysis 


¢ OPM measures are typically a 
subset of the program’s measures 

e Not all QPM measures are 
Statistically managed 


Key to understand the difference 


between quantitatively and 
Statistically managed measures. 
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OPP and QPM Interaction 


Goals and objectives 


Goals and objectives 


Measures 
Goals & Obj. / ae 


Measures 
Models 
Goals & Obj. 


Org Capability 


Org Analysis 
of Program Data 


Organization Org Capability 
Measurement and Analysis 


Repository 


Program M&A Data 
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5 Organizational Innovation and Raytheon 
Deployment (OID) 


e Builds on the Organization Process 
Focus (OPF) foundation 


e Improvements are driven by OPP 
and QPM activities (org and 
program) 

e Effect on the organization’s 
quantitative goals and 
objectives are measured 


e A Level 5 organization is a 
learning organization 


OID process 
improvement 
Innovations 

(tied to organization’s 
quantitative goals and 
objectives) 


This is a tie to Level 4 


Link to the organization’s 


quantitative goals and 
objectives IS key. ly 2006 . All rights reserved. 
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5 Causal Analysis and Resolution Raytheon 
(CAR) 


e CAR process can be applied 
regardless of the 
organization’s maturity level 


e CAR process maps well to the 
Raytheon Six Sigma™ 
process 


¢ Not all R6o® projects meet the 
CMMI Level 5 model 


e Level 5 R6o® projects are 
driven by the program’s 
quantitative goals and 
objectives 

Link to the program’s 


quantitative goals and 
objectives is key to Level 5. 
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SAS Program Representation of  Baytheon 
Level 4 and 5 


Org 
Defined 


Plan to collect 
metres Program M&A Plan 


Collect 
Set program Metrics 


objectives/goals 


Process 


Metrics Analysis Meetings Review 


és Becags Metrics at 
Analyze metrics quantitatively Program 
¢ Compare to org norms 


Meetings 
¢ Use predictive models 


; I ¢ Determine if objectives can be met 
Learning 


cee | 
Organization 
[Para] 
Defect SPC Analysis 


Metrics 


Organization 
Repository 


~ Root Cause 


: » Analysis & 
R6o Corrective Yes 
Rayon Si Sigma Action 


Negative 
Trending 
or Outlier? 


Logger 


No 


Best Practice/ 
Lessons Learned? 
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Using the information from the Raytheon 
Level 4 and 5 model review 


Use the documented understanding of Levels 4 and 5 


e To educate and inform 
— EPG members 


— Persons collecting evidence 
for a SCAMPI 


— Program personnel involved in the 
Level 4 and 5 activities 

To relay the organization’s 

Level 4 and 5 processes to 

the appraisal team members 

— Through meetings and discussions 


— Through organization and program 
demonstrations of the Level 4 and 5 activities 
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Raytheon 
Summary 


= A Level 4 or 5 organization must be built on an established 
Level 3 foundation 


= Consistency in CMMI interpretation is essential to a 
successful appraisal 


= The organization must understand and be able to present 
how it is meeting the intent of the practices 


= Show how you make the pieces fit together! 
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Raytheon 
Questions 
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Raytheon 
Contact Information 


e Nancy M. Raymond 972.344.5451 
n-raymond@raytheon.com 


e Linda A. Kovar 310.334.1828 
lkovar@raytheon.com 
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Raytheon 
Biography 


e Nancy Raymond is the Deployment Support Cross Product Team Lead in the 
Space and Airborne Systems Enterprise Process Group. Nancy has been with 
the company for 28 years. For most of her time at Raytheon, Nancy has beena 
software and process engineer on programs and within the organization. Nancy’s 
specialties include software planning, estimation and metrics. Nancy has been 
involved with the development and deployment of CMM/CMMI compliant 
processes for more than 10 years. Nancy has a Masters Degree in Mathematics 
from Carnegie-Mellon University. 


e Linda Kovar is a Senior Manager of Programs at Raytheon Space and Airborne 
Systems in El Segundo, California. She is the Enterprise Process Group (EPG) 
lead for their process improvement activities. In August 2005, she successfully 
led the activities in achieving a combined Systems, Software and Hardware CMMI 
Level 3 for the business unit which encompasses 7000 engineers in several 
states. She is currently leading the Enterprise Process Integration activities to 
focus on a CMMI Level 5 maturity across all of SAS Engineering. Linda has 25 
years with Raytheon where she has held positions in Program Management, 
Functional Management, Technical Staff, Test Program development and 
software engineering. She is a certified Raytheon Six Sigma expert, has a BS 
degree in Computer Engineering from the University of California Los Angeles 
and is a graduate of the Anderson School of Executive Management. 
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National Defense Industrial Association 


6th Annual CMMI Technology Conference 
November 2006 


The Value of Systems 
Engineering 


What do we know about it? 
How do we discover more? 


Al Mink 
Systems Value / GMU 


Value of SE 


Overview 


1. The Problem 
2. What We Know Today 
3. The Race to Discover More 


4. Conclusions 


SSS SSS ————————————————————————— ary 
Al Mink Value of SE - NDIA 2 


Value of SE 
The Problem (Stakeholder Analysis) 


=» Customers 
= Unsure of how to evaluate bids 
= May not receive best value for the systems they acquire 
= DOD #1 SE Issue - “Inconsistent SE Practices across life cycle” 
= Industry (System Developers & Integrators) 
= Unsure of what to bid, and later loath to add SE costs 
= Associations & Academia 
= Unable to fully satisfy their members and students 
= SE professionals 
= Lack rigorous justification for their recommendations 


Al Mink Value of SE - NDIA 3 


Value of SE 
The Problem (IDEF 0 View) 


Inputs of Stakeholders 


Operational 
ve System 
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Value of SE 
The Problem (Pareto View) 


Cost 
a ae ae 
Cost Committed 
UY Fe pitas ermine on Cost Incurred 
60% 
ee 
i | 
40% | | 
: : 
20% — ama aaa? 
| \ 
i ; 
0% ee ——a 
Conceptual Detailed Construction Use, 
& Preliminary Design & or Refinement Buede, Figure 1.2 
Design Integration Production & Disposal 
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Value of SE 
What we know today —- Studies & Models 


Gruhl, National Avionics and Space Administration (NASA), 1992 
Compared upfront expenditures to eventual cost growth 


Herbsleb, Software Engineering Institute (SEI), 1994 
Studied ROI on process improvement in software 


Honour, International Council on Systems Engineering (INCOSE), 2002 
Surveyed industry to compare SE Effort to cost & schedule 


Boehm & Valerdi, SE ROI (COCOMO), 2006 (Draft) 
Analyzed SE activities from COCOMO II 


Valerdi & Boehm, Constructive System Engineering Cost Model 
(COSYSMO), 2004 
Developed parametric estimation model similar to COCOMO 


Others... 
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Value of SE 
What we know today —- NASA Study 


Source Werner Gruhl 
NASA Comptroller’s Office Total Program Overrun 


& Honour 2004 32 NASA Programs 


Definition $ 
Definition Percent = ---------------------------------- 
Target + Definition$ 


Actual + Definition$ 
Program OVEIIUN = ------------------------0-n---=--= 
Target + Definition$ 


Diminishing Returns 
SEASAT $_anas 
o O 


-_ UanS® EUVE/ EP R? - 0.5206 
Sd 


ULYS 
PIONVEN 


= 
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= 
o 
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= 
eS 
D 
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= 
O 


. 
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Definition Percent of Total Estimate 
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Value of SE 
What we know today -— INCOSE Study 


3.0 Source: 


Honour 
2004 


2.6 


2.2 


1.4 j Diminishing Returns 


2 = 


Actual/Planned Cost 
fee) 


ball J = = 
- 
S23 es es =e oe 
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SE Effort = SE Quality * SE Cost/Actual Cost 
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Value of SE 
What we know today - ROI of SE 


PPT eee 


Boehm & Valerdi, 2006 
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Value of SE 
What we know today - ROI of SE 


100 s 

® 80 

O 

O 

TO 

G 60 

5 

< 40 

> 48 Development Cost Increase: 

20 Minimal SE compared to Robust SE 
10 100 1,000 10,000 
Boehm & Valerdi, 2006 Software Product Size (KSLOC) 


Al Mink Value of SE - NDIA 10 


Value of SE 
What we know today - COSYSMO 


Size 

Drivers Person 
Months of 

Effort — systems 

Multipliers engineering 
effort 

Pred(30) 50% uncalibrated 
Pred(30) 70% calibrated 
Valerdi, 2005 
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Value of SE 
What we know today - COSYSMO 


7% E 
30% 
31% 


Valerdi & Wheaton 2005 
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Value of SE 
What we know today - COSYSMO 


Operate, 
Maintain, 
or 

Enhance 


Replace 
or 
Dismantle 


Valerdi & Wheaton 2005 
Al Mink 
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Value of SE 
What we know today - COSYSMO 


Operational Transition to 


Conceptualize Develop Test & Operation 
Evaluation 


Ww : " y= nr 
enncer Lcoeo 
SUil EC 1JZ00 


Acquisition & 
Supply 


Technical 
Management 


System 
Design 


Product 
Realization 


Valerdi & 
Wheaton 2005 


Technical 


Evaluation 
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Value of SE 


What we know today - Summary 
STUDY APPLICABILITY 
Author & aoerae eee Definition of | Characteristics 
Findings SE Activities . 
Background Success of Project 

Gruhl (1992) 8-15% Upfront ] First two of five Cost (Less cost Large; Complex; all 
32 NASA Pgms Best development phases | overrun) NASA 
Herbsleb (1994) Process CMM Process Cost (Cost Vaidouss federal 
13 CMM Improvement oo reduction through bis eA 
Companies ROI 4.0 — 8.8 SE investment) 8 
Honour (2004) 15-20% of Overall SE level of Various sizes 
Survey INCOSE project should effort (Cost) & Cost & Schedule (measured by total 
SEs be SE related SE quality project cost) 


Boehm & Valerdi | SE importance | COCOMO II RESL Various sizes, but 
(2006) grows with (Architecture and Cost software systems 
COCOMO II project size Risk) only 


: Estimate 
Bochner ee Valeray |} cithin 300% 33 activities defined Mostly swccessttn 
(2004) effort 50% - by EIA 632 Cost projects from 
COSYSMO is y federal contractors 
70% of time 


Ancona & Team boundary Product 


Caldwell (1990) a. — a He activities — interface | Performance Technology 


Boundary eae = ence between team and (Successfully products 
Management external marketed products) 
More SE Product Three similar 
Frantz (1995) yielded better Performance & systems for 
Boeing side-by- quality & Defined by Frantz Schedule (Quality manipulating 
side projects shorter of product and airframes during 
duration duration of project) assembly 
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Value of SE 
What we know today ~ Summary 


8-20% 
in 
Aggregate* 


Cost, 
Relative to 
budget 


* COSYSMO tracks 33 SE activities 
a eS SS SSS Sag SSS SS ne 
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Value of SE 


The Race to Discover More 


SYSTEMS AND 
SOFTWARE 
CONSORTIUM 


Carnegie Mellon 
Software Engineering Institute 


SUS Sul! 


CONSTRUCTIVE SYSTEMS ENGINEERING COST MODEL 


l hen. 


University of 
SEEC South Australia 


E 
| eee Inc. __ 
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Value of SE 


The Race to Discover More 


By SE 
activity 


Al Mink Value of SE - NDIA 18 


Value of SE 
The Race to Discover More - Methodology 


|_All Four Appear to Follow a General Approach _| 
. Form Team 
. Develop Approach 
. Identify Projects 
Collect Data 


. Analyze Data 
. Publish Results 


ou BR WN PB 
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Value of SE 
The Race to Discover More - Methodology 


Limitation of Project Informatiddata Protection Guidelines 
Interests of Stakeholders Resource Limitations t jations oF Collection ities of Project Owners Publication Policies 


Project Charter 


Data Collection Approact) 


Project Information 


Factors - Value of SE 


Factors 4yailable to SEs 


Sponsoring Organization Research Team Project Owners Data Collection Mechanism Analysis Team Media Vehicles 


Date: 
Saturday, April 29, 2006, 


(Academic) Determine value of SE 
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Value of SE 


The Race to Discover More 


Sensitivities of Project Qwners 
Data Collection Approach 
Invitation to Participate 
Limitation of Project Information 
Limitations oF Collection Mechanism 


Project InFormation Date - Project Characteristics 


4.2 
Measure 
Success 


 Oasts - SE Efforts 


4.3 
Characterize 


Date - Project Success 


Dats Collec 
Data Collection - 5E & 


Data Collection — Project 


Date Collection Mechanism 


Date: 
Saturday, April 23, 2006 


4 
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Value of SE 


The Race to Discover More - Define SE Activities 


Buede 
pg 416 
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Value of SE 


The Race to Discover More - Define SE Activities 


Defining “SE Activities” - Many Views 


Fragmented by 
rolttolse)itar=me) dialer 


Fragmented by 
ofelsir-liame) *)ialcelat 


Fragmented by 
ie Talet-lae ts 


Honour 
2005 
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Value of SE 


The Race to Discover More - Define SE Activities 


| Standards 


NATIONAL DEFENE 
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Harnessing complex 
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CONSTRUCTIVE SYSTEMS ENGINEERING COST MODEL 
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Value of SE 


Emerging Approaches to Move Forward —- Define Other Measures 


Cost, 
schedule, & 
quality 


Project characteristics 
Success factors - Size ($) 
- EVMS - Size (hours) 
- Award Fee - Technology 
- Requirements Trace - Complexity 
- Others... - Others... 
Al Mink 
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Project Information 


Interests of Stakeholders 


Project Charter 


Value of SE 
The Race to Discover More - Methodology 


Limitation of Project Informatiddata Protection Guidelines 


Resource Limitations 7 ations oF Collection Meskanit 


Data Collection Approact) 


Sponsoring Organization 


Research Team 


wacter|stics 


ties OF Project Owners 


Factors - Value of SE 


Analysis Team 


Publication Policies 


Media Vehicles 


Factors 4yailable to SEs 


(Academic) Determine value of SE 
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Value of SE 


The Race to Discover More - Define SE Activities 


Data Collection 


Harnessing complex 
socio-technical Sysfems 


SUN Ee 


CONSTRUCTIVE SYSTEMS ENGINEERING COST MODEL 
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Value of SE 


Conclusions 


= Value of SE 
= Remains fundamental to furthering SE as a respected discipline 


= Four approaches underway to determine SE Value 
= With a fifth - Bob Bruff — on the horizon... 
= They share commonalities, but also differ: 


= Differing types of projects 
= Differing SE Activities & Deliverables 
= Differing success factors (cost, schedule, quality, etc.) 


a Challenges Remain 
= Useful project data - may not be widely available 
= Four separate projects — what if they report different results? 
m™ Success may be elusive - “The Shangri-La of ROI” (Sheard 2000) 
= Make a difference! Support these approaches 
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Value of SE 


Conclusion 


VT oe 
almink@systemsvalue.com SYSTEMS AND 


SOFTWARE 
CONSORTIUM 
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Harnessing complex 
socio-fechnical Systems 


CONSTRUCTIVE SYSTEMS ENGINEERING COST MODEL 


Ricardo Valerdi 
rvalerdi@mit.edu 


Eric Honour 
ehonour@hcode.com 
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Value of SE 


Questions? 


Al Mink 
almink@systemsvalue.com 
571 212-4778 
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Evaluating the Impact of 
The QuARS Requirements 
Analysis Tool 


Using Simulation Nasal 


David M. Raffo, Ph.D., Portland State University 
Robert Ferguson, Software Engineering Institute 
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Agenda 


Motivation 

What is QUARS? 

What is Process Simulation? 
What are the Benefits? 
Discussion 

Conclusions 
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Motivation 


Good new technologies are wasted unless there is a 
compelling business case to use them 


Without such a case: 
¢e Managers not convinced 
e No reallocation of scarce resources 


Good technology: QUARS Requirements Checking 

Tool 

e Increased PDs (probability of detection) (enables 
better detection capability during human inspection) 

e Low cost 


This talk: 
¢ Present the business case 
¢e Developed using process simulation 


Portland State 
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Requirements Best Illustrate Our Challenge 


Distribution of Effort 


Distribution of Defects to Repair Defects 


Code Code Other 

7% 1%. 4% 
Other Requirements ln 

10% 82% 


Requirements 
| 56% 


Design 
13% 


Design 
27% 
Over half of software defects are Over 80% of rework effort is spent 
attributed to requirements problems on requirements-related defects 


Source: James Martin Source: Dean Leffingwell 
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Analyzing Requirements, 


An endemic and enduring problem 
e Vague requirements with unstated performance 
criteria 
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QuARS: A part of the solution 
¢ Quality Analyser for Requirements Specification 
e Lexical, and syntactic analyses of requirements 
documents 


Uses: 

¢ Real-time editing of requirements defects 

e Inspections and quality assurance 

¢ Tracking and improvement of requirements 
analysis processes 

¢ Contract acceptance and appraisals 


gf, Portland State 
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Analyzing Requirements, 


Why use it? 

e Reduce cycle time and effort while producing 
better results than possible with tedious manual 
review 

e Early detection and correction of often costly 
errors 
- Captures most common classes of errors 
- Often missed in inspections and quality 

assurance 
- Allowing analysts to focus on more difficult 
problems 
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Analyzing Requirements, 


How does it work? 
e Natural language analysis of requirements text 


¢ Lexical: vague, weak, optional, subjective, other 
terms 
e Syntactic: multiple, implicit, under specified 
statements 
¢ Semantic: 
- Allows screening for consistency, 
completeness, etc 
- Arbitrary combinations of domain, component, 
functionality, product quality attributes and so 
on 


Software Engineering Institute 


Fle dt “Wie Analysts Tooke 
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The Hine umber: 
T7231 For this pupase. the rcewer wll manage @ table af evuabled tranamttore 
contains an implicit sentence (implicit 


The fine number: 
T1241. The inqpection inatrument sade the bay code and sevads the data to the NBG. which stove tien | || 


contains an implicit sentence (implicit object). 


The er: 
|] 281 6 Stepe 2 though 5 are repeated for each remate radio tranamitter hey that must be loarned, “3 
‘| 


QuARS 


165 1,442.7 Rofing cose algomhm 


188 The rating code fonction is implernunted using the KiXwlog(t) algortin The supplier shall guarantee that the algarthm « consistent with the requirments of EC Dire 
18 
170 A detaded description of secem code and data managemerd is provided m the documont “Veticie Proiachan System Secret Codec” 
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Syntax Parser 


Indicator-related 
dictionaries 


Parsed 
Requirements 
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Syntactic |>> >> | graphical 


Analysis | indicators 
i « Logs 


Semantic derivation | ___ > : 


Domain 
dictionaries 
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What is Process Simulation* 
¢ Process simulation models focus on the dynamics of 
systems development, maintenance and acquisition projects 

e They represent the process 

- aS currently implemented (as-is, aS-practiced, as- 

documented), or 

- as planned for future implementation (to-be) 
e Simulation Features 

- Use Graphical interfaces 

- Utilizes actual data/ metrics 

- Predict performance 

- Supports “What if’ Analyses 

- Support business case analyses 

- Reduces risk 
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Applying Process Simulation = High Value Add 


Evaluate Strategic Issues 
¢ Quality Assurance, V&V and IV&V Strategy 
¢ Distributed Software Development 
e Supply Chain Design 


Plan Processes 
e Identify better process alternatives 
e Assess the Costs and Benefits of New Tools 
e Evaluate Impact of Process Improvements 


Architect, Design, and Document Processes 
Manage Projects Quantitatively 

Estimate Project Costs from the Bottom Up 
Train Project Managers 


zee, Portland State 
wg University 
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How do we use Process Simulation? 


Architect the Process Model 
Calibrate the Data Set 

Run Options 

See the Return on Investment 


Portland State 
University 


odels 


Process Simulation Mode -e 
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Management 
Dashboard 


Project is 
Approve 


a 


° IEEE nae Generalized Process Components : Development 
e Spiral e Inspection 


¢ Incremental Reqi: Use Case Analysis ¢ Testing 
e Product Line e Rework 
° Rapid Prototyping = oo = _ e IV&V 


eJoint Reviews 
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Development Projects 


ICSCI Data (Follows) 
INo. of CSCIs 8 
ICSCI names: C&DH Guidance & MEPS Ground DIVINERLAMP LOLA ~~ LROC 
Estimated SLOC 
Reuse Re-eng New Lang Totals Ptotals IVVTotals Ee ce 


25000 75000 100000 120000 150000 
25000 75000 100000 120000 1500003 


eOrganizational 
eSite and Project “> 
e Industry Standard 


Project 
Database 


SW Process Simulation Model oo 


5800 43135 1056858 
163 1434.75 21746 
aR00 53 AGB RS O87317 
163 151191 29802 
ROO BDH TOSSES 
163 1434.75 21746 
ROO Boma 004343 
163 1445.0 20273 
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NASA Model - Includes IV&V Layer 
with IEEE 12207 SW Development LC 


Joes 


Wcount Entities : Soxx : : q a: 
ee ed —el 


Results: 
Unit 
Se x 2 KSLOC 

Effort . Person Months 
Rework Effort . Person Months 

Duration 5 Months 
Avg. Duration . Months 
Inj. Defects 
Det. Defects 
Time Time Cor. Defects 
— Requests ane Y2 Utilization wm, Numbe| ann YZ Utilization «om Numb Latent Defects 
we Black 


o i) 
B197.792 { 8197,712 


— 
—_——-. 
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System and Software Requirements 
Processes 


AS-IS 


TO-BE 
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IV&V at Requirements Verification 
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Impact of QUARS - Assumptions 


Have the ability to look at a variety of process 
improvements 


Assumptions: 
¢ Typical Manned Mission using IEEE 12207 Process 


e Includes IV&V 
¢ 100 KSLOC Project 


e Industry standard data for Earned Value, defect 
detection rates 


¢ Organizational data for productivity, defect injection 
rates 


e Project specific data for IV&V 
e Pilot study data for capabilities of QUARS 


Portland State 
University 


Impact of QUARS - Axeumplions 
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In the case of QUARS 
1. Productivity of the tool => 10 KLOC/ Person hour 


2. QUARS type defects => 37% of Requirements Defects 
3. QuARS detects 100% of lexical and (i.e. QUARS detectable defects) 
4 


Improves defect detection capability at Requirements Inspections (+5 
to10%) 


5. Cost of training and associated SEPG activities 1 person-month 
6. Cost of tool TBD 


Secondary Effects of Using QUARS 
1. Improves clarification of requirements (i.e. improves productivity in 
design of + 5% 


2. Improves Engineering design decisions (reduced injection of design 
defects of - 5% 


3. Improves test planning and test case generation productivity + 5%) 
4. Improves test case generation (i.e. less investigation and rework -5%) 


gf, Portland State 
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Cases Looked at 


QuARS as a V&V activity within the project. 
¢ Look at applying QUARS at the Systems Requirements 
and Software Requirements phases, both. 
e Assuming 100% and 50% Requirements inspections 
¢ Before and after inspection 
e When injection of QUARS type defects is at minimum 
(i.e. 20%) 


QuARS as an IV&V activity outside of the project 

¢ Look at applying QUARS at the Systems Requirements 
and Software Requirements phases, both. 

e Assuming 100% and 50% Requirements inspections 

e When injection of QUARS type defects is at minimum 
(i.e. 20%) 


— 
——_.- 
——_ — 
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Key Questions Evaluated 


Did QUARS provide a value? 
Is the tool more effective in V&V or IV&V mode? 


Under what project conditions is the tool most useful? 
¢ Applying QUARS before or after Requirements 
Inspection 
¢ Applying QUARS when different amount of requirements 
are inspected 


Is QUARS still worth using when lexical defects are ata 
minimum? (max reduction through training achieved) 


What is the amount that NASA should be willing to pay for 
the tool? 


Portland State 
University 


Results - Applying QUARS in V&V 
mace # at Different Phases 


Comparison to Baseline 
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QuARS at Sys Reg 

p value 

QuARS at Sw Reg 

p value 

QuARS at Sys & Sw Reg 
p value 


e Application of QUARS at Systems and Software Requirements 
offers a value 

¢ Sweet spot is to apply QUARS after Software Requirements 

¢ QUARS Is approximately +10% to +15% benefit when applied 
before Requirements inspection rather than after 

¢ QUARS has approximately +3% increased performance when 
project does not have IV&V 


Portland State 
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Inspected 


Comparison to Baseline 


Effort | Rwrk_Efrt 
2165 1600 
0 


6576 
0 
6340 


e The value of QUARS increases when applied to projects that 
experience less than 100% inspections (this instance = 50%) 


e At 50% inspection, +20% to +30% increased effort savings, 
+17% to +%42% reduction in latent defects 


$2, Portland State 
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Results - Applying QUARS in IV&V 
Mode at Different Phases 


Comparison to Baseline 
Effort incl. V&V 
Wav Effort = =Rwrk_Efft Effort Duration Avg. Dur Cretd_Dfcts Ltnt_Dfcts 
QuARS at Concept IV&V 1,446 1,679 1,322 (231) 114 69 32 it 
p value’ 0” 0’ 0” 0” 0” 0” i 0 
QuARS at REQ IV&V 2427 2017 2,341 (290) 191 b4 19 24 
p value’ 0” 0” of 0” 0” of ad 0 
QuARS§ at both IV&V 2,900 3,374 2,976 (474) oat 98 11 36 
p value 0 0 0 0 0 0 1 0 


Value of QUARS is significantly reduced when applied in 
IV&V mode. 87%, 47%, 55% for effort; 94%, 52%, 61% 


Secondary effects not experienced by the project 


Slight make up on effort due to cost shift to IV&V 
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Defect Injection Rates 


Lexical defects reduced from 37% of Requirements defects 
to 20% (46% reduction) 


Believed that even with training and other defect prevention 
measures, lexical defects will still exist at 20% level or 
greater 


For V&V 
e Effort savings reduced by 28% to 36% 
¢ Quality savings reduced by 28% to 38% 


For IV&V 
e Effort savings reduced by 35% to 43% 
¢ Quality savings reduced by 26% to 36% 
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Results — QUARS Under Different 
Defect Injection Rates 


Comparison to Baseline 
Effort incl. V&V 
VaV Effort  Rwrk Eft Effort Duration Avg. Dur Crcetd Dfcts Ltnt Dfcts 
QuARS at SysReq 1,186.78 1,199.64 958.23 (1286) 5817 42.36 39.21 13.08 
p value” 002" 002" OOO” o84" 0257 0.087 051" 0.04 
QuARS atSwReq 3,179.53 3,187.55 2,990.71 (8.02) 212.62) 44.53 13.00 34.37 
p value” 000" ooo" ooo" O90" OOO” 005” 083" 0.00 
QuARS at Sys&SwReq 3.35404 3,295.10 2,994.34 56.94 235.76 62.62 13.69 36.25 
p value 0.00 0.00 0.00 0.33 0.00 0.01 0.92 0.00 
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_|Comparison to Baseline 
Effort incl. V&V 
Wav Effort = Rwrk Eft Effort Duration Avg. Dur Cretd Dfcts Ltnt_Dfcts 
QuARS atConcept IV&V 874.24 «1,174.63 © 833.20 (300.39) 1748) 4467) 36.16 = 12.66 
pvalue 008° 0.027 000° 000° O71" 008° O54" 0.05 
QuARS atREQIV&V 1,571.89 1,747.96 1,396.61 (176.07) 12347 4522, 30.78) 17.48 
pvalue’ 0.00' 000° 000° 000° O01 006" 060° 0.01 
QuARS atboth 1,643.16 2,123.36 1,758.96 (480.20) 86.25 6177 2449 23.22 
p value 0.00 0.00 0.00 000 O07 0.01 0.68 0.00 
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Return on Investment Inputs 


Input Parameters for Financial Calculation 


Input 
Cost of Development Staff per Hour $ 100.00 
Cost of IV&V Staff per Hour $ _ 100.00 


Org internal investment rate cut-off 00% 
(aka hurdle rate) a 
Implementation Cost (Tool Cost) $ ~ To be determined 
Increase in Revenue per Month $ - if release early 


Cost to Correct Latent Defects $ 25,500.00 per defect 


1.5 person-month to fix 1 defect 
Assumptions 
Work Hours per Month 170 Work Hours per Year 2,040 
Latent Defects will be corrected a nonths 
within the first 
If releasing the system early by 3 months or more, there will be an increase in revenues (due to ar 


Effort saving occurs at time = duration 
Duration saving occurs at time = duration+ 1 month 170 hours 
Latent defect saving occurs at time = duration + 36 month 6120 hours 
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$282,813 $202,714.31 | 25,167.62 
$869,221 $539,454.13 | 30,814.83 


$933,118 $576,941.36 

$265,149 520.884 $182,139.63 | 16.474.07 | 
$837,796 $270,598.43 
$876,587 $335,364.05 
$266,008 $19,413 

$435,653 $29,121 

$541,538 $34,524 


¢ PR(NPV>0)=100% 

e PR(NPV>$100K) = 100% 

- Overall, QUARS shows a reduced NPV between 
-28% to -38% compared to higher defect injection 
rate (Lowest NPV = $182k) 
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Scenario Descriptions 


SC 1: Turn on QuARS at System Requirments 

SC 2: Turn on QUARS at Software Requirements 

SC 3: Turn on QuARS at both Systems and Software 

SC 4.1: Turn on QuARS after System Requirments Insp 

sC4.2 Tur on QUARS affer Software Requirments Insp 

C43: Turn on QUARS affer both System and Software Requirments Insp 
5C6: Turn on QUARS at Concept Verification 

SCT: Tum on QUARS at Requirements Verification 

C6: Tum on QUARS at Concept Verification and Requirements Verification 
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Discussion 


Straight forward and quick analysis (1 week) 
¢ Main effects analysis 
¢ Secondary effects analysis 
e Sensitivity analysis 
e Management Questions 
¢ Results 


NASA is currently engaged in conducting a 6 month trial of 
three different requirements analysis tools 


Will use results of their study to validate the model 


Still need to run simulation model to compute overall 
impact of the tool and perform business case analysis 


gf, Portland State 
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Conclusions 


QuARS is worth while 
¢ Value to the project @ 20% hurdle rate ranges from 
$280K to $930K in V&V mode and $266K to 540K in 
IV&V mode 
¢ Cost of tool is not set yet 
¢ PR(NPV>100K) = 100% 


Analysis showed that results were sensitive to 
¢ % of project inspected 
¢ % Lexical defects injected 
e Labor rates, rework costs, hurdle rate 


For these parameters, it is important to be clear about their 
values for projects that NASA plans to implement QUARS to 


Straight forward analysis took about 1 week. 
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Conclusions 


Process Simulation is NOT a Silver Bullet 


Many High Value Add Ways to Use Process Simulation 


e Evaluate Strategic Issues - Quality Assurance Strategy 
e Plan Processes 


- Assess the Costs and Benefits of New Tools 
e Architect, Design, and Document Processes 


e Manage Projects Quantitatively (CMMI L4) 
e Estimate Project Costs from the Bottom Up 
e¢ Train Project Managers 


See SEI Technical Report on Transitioning Process 
Simulation into Organizations (Spring 2007) 
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Contact Info 


David M. Raffo, Ph.D. 

Visiting Scientist, Software Engineering Institute 
Associate Professor, Portland State University 
Principal, Quantel, Inc. 


raffod@pdx.edu 
Cc) 503-939-1720 


Robert Ferguson 
Software Engineering Institute 


rwt@sei.cmu.edu 


Portland State 


=— Carnegie Mellon rtlanc 
University 


~——— Software Engineering Institute 


The End 
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Understanding and Leveraging a Supplier’s 
CMMI Efforts: 
A Guidebook for Acquirers 


Lawrence T. Osiecki 
Software Engineering & System Assurance 


OUSD(AT&L) SSE/ SSA 
15 November 2006 


Briefing Outline 


e Background 

e Guidebook need 

¢ Content Authors 

e Issues 

e Published Maturity Levels 
e Guidebook Contents 

¢ Format of the Guidebook 
¢ Current Status 

e Path Forward 

e¢ Summary 
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Background 


NDIA hosted “Use of CMMI in Acquisition: Summit & Workshop” in 
Sep 05 
Purpose of the Summit was to look at a set of existing issues 
(representative): 

—- Background on how organizations approach CMMI appraisal and why 

— Use of the Appraisal Disclosure Statement 

- Formal Guide for CMMI Usage for DoD 

- Training for DoD Acquisition Organizations in the use of CMMI 
Findings (Action Items) resulting from the Summit included (not 
comprehensive): 

—- Complete Guidebook for use of CMMI in Acquisition 

—- Recommendation for Guidebook topic coverage 

- Generate suggested input to RFP section lL & M 
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Background - 2 


The Summit Results suggested the following Guidebook topics: 


Interpreting the Appraisal Disclosure Statement (ADS) 
List sources for help (FFRDCs, DCMA, others) 
Generate suggested content for RFP Sections L & M 
Background/ description of CMMI esoteric content/issues 
Getting processes transitioned/implemented rapidly on new programs 
Processes in the |MP/IMS/SOW (achieving contractual commitment to 
execute) 
Offeror approach to verifying compliance to proposed processes 
Past performance measures to provide evidence of process 
performance 
Appendices: 
¢ Checklists 
¢ Contract Language 
e RFP language 
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Content Authors 


¢ OSD and the CMMI Steering Group (CMMI-SG) identified this gap 


e The CMMI-SG chartered a committee of Government, Industry, and 
Academic contributors to develop guidance for acquirers 


Hal Wilson (Northrop Grumman Corp) Joe Wickless (SEI) 


Linda Rosa (Mitre) Mike Phillips (SEI) 
Mike Nicol (ASC/EN) Brian Gallagher (SEI) 
J eff Dutton (J acobs Engineering) J oe Elm (SEI) 


Gene Miluk (SEI) 


¢ Government Team is coordinating the final effort 


— taking this body of work and structuring it in Guidebook format for use 
by Program Managers 
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Issues 


CMMI-DEV in an acquisition environment 


Intended Use: CMMI (the original CMMI model) is designed for use within 
development organizations for internal process 
improvement, not from acquirers perspective 

Size: CMMI is a comprehensive process model that exceeds 700 
pages. Just another tome the PM must have knowledge 
on how to implement 

Training: CMMI training focuses on the use of CMMI-DEV for 
process improvement, not leveraging the activities from an 
acquirer’s perspective 

Marketing: “level-mania” too often has defeated desired behavior of 
deploying process improvement at program startup and 
throughout the development life cycle 


Guidance in the effective use of CMMI-DEV by 


Acquirers does not exist 
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Issues -2 


e These Issues lead to difficulty in: 


— understanding what specific benefits CMMI provides for the acquiring 
program office 


— understanding how CMMI reduces program risk 

— determining what to ask for regarding process capability 

— interpreting developer’s claims regarding maturity level ratings 

- discriminating between the process capability of competing suppliers 


—- separating the “practitioners in the marketing department” from the 
“oractitioners on the shop floor” 


— monitoring developer’s process capability and their implementation as 
applied to current programs 


—- understanding how broadly the rating applies to the organization of 
interest to the acquirer 


- belief in the myth that CMMI is the “silver bullet” 
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Published Maturity Levels 


e Published Maturity Levels may be based on a single 
Division or location 
XYZ Corp. 


T walold 
Lwpaloid 
A yaloid 


MLs usually 
apply HERE 
based upon 
appraisals of 
THESE ... 


... but your 
project is 


HERE or HERE 


The new ADS will help clarify these claims 
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Executive Summary 


e This Guidebook intends to help acquisition 
organizations understand and apply CMMI -DEV 
In three ways: 


— to provide guidance on how to request, interpret, and 
use the results of CMMI appraisals 


- during source selection, to help identify and evaluate 
strengths, weaknesses, and risks that may be 
associated with a particular offeror/team approach 

— throughout the execution of an acquisition program, 


to gather and interpret information for effectively 
monitoring contractor processes 


Provide insight to what CMMI Is... and Is not 
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Guidebook Contents 


e The guidebook provides the basic information that 
acquirers need in order to understand CMMI use by 
Suppliers 

- explanations of capability and maturity levels 

— differences between continuous and staged model representations 
— high maturity/capability levels (levels 4 & 5) 

— addresses obscure elements of CMMI, such as equivalent staging 


e Cautions that high capability and maturity level ratings do 
not of themselves guarantee program success 


e This Guidebook intends to help clarify what high capability 
and maturity ratings can and cannot do for a program 


Slide 10 
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Guidebook Chapters (draft) 


Introduction 
Understand the Development Process Needs for the 
Program 


Understand the Process Capabilities 


of Potential Suppliers 


Structure the Contract and Award Fee Criteria to Address 
Process Risks 
Monitor Supplier Process Performance After Award 
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Guidebook Chapter 1 (draft) 


e |ntroduction 
— CMMI Fundamentals 
— CMMI Appraisals 
— Interpreting CMMI Ratings 
— The Structure of the Guidebook 
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Guidebook Chapter 2 (draft) 


e Understand the Development Process 
Needs for the Program 
— Estimate Exposure to Process Risk 
— Build Process Capability into the Program 
— Relate Processes to the Program Life Cycle 
— Assess the Process Needs of the Program 
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Guidebook Chapter 3 (draft) 


e Understand the Process Capabilities of Potential Suppliers 
— Request Recent Appraisal Results 


— Request Integrated Master Plan Documentation of Proposed 
Processes 


— Request Process Proposals in Critical Areas 

— Request Approach to Integration with Subcontractor Processes 

— Request Approach to Integration with Acquirer Processes 

— Request Historical Data on Process Performance 

- Evaluate Compliance of CMMI Process Deployment to |!SO 9001-2000 
— Use Recent Post Award Appraisal Data from other Programs 


—- Request Incorporation of Process Reviews into Proposed Program 
Schedules 


— Enable Process Compliance Evaluation During Execution 
— Perform Pre-Award Process Appraisals 
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Guidebook Chapter 4 (draft) 


e Structure the Contract and Award Fee Criteria to 
Address Process Risks 


— Incorporate Process Reviews into the Program 
Schedule 


— Specify Post Award Appraisal 

— Specify Process-Based Award Fee Determinants 
— Require Risk Mitigation Plans and Activities 

— Specify Reporting of Process Statistics 

— Specify Access to Selected Process Artifacts 


— Approaches when Process Capability not Addressed in 
the Solicitation 
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Guidebook Chapter 5 (draft) 


¢ Monitor Supplier Process Performance 
After Award 
— Process Monitoring through Artifact Review 


— Process Monitoring through Review of Process 
Metrics and Audits 


—~ Contractor Team Process Reviews 
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Guidebook Appendices (draft) 


e Appendix A - Questions & Checklists 

e Appendix B - Understand the Development Process Needs of the 
Program (Implementation Guidance) 

e Appendix C - Understand the Process Capabilities of Potential 
Suppliers (Implementation Guidance) 

e Appendix D - Structure the Contract and Award Fee Criteria to 
Address Process Risks (Implementation Guidance) 

e Appendix E - Monitoring Supplier Process Performance after Award 
(Implementation Guidance) 

e Appendix F - SCAMPI-A Appraisal Disclosure Statement (ADS) 
Example Template 

e Appendix G - Acronyms 
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Format for the Guidebook 


e Readable (small) 40 pages for the PM 


e Electronic delivery of additional material 
designed for implementation 


e Linked appendices for online version 


e Printed version (40 pages) with CD or with 
linked appendices 


e Part of the CMMI Product Suite 
e |ncorporate into DAU training 
e Update as our learning occurs 
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Current Status 


¢ Committee chartered by CMMI Steering Group 

e First draft completed and submitted for Steering Group 
review (Jj ul 06) 

e Second draft completed and submitted for government 
review (Aug 06) 

¢ Government team assumes leadership role for 
Guidebook completion (Oct 06) 


e Guidebook content being restructured based on 
Government reviews on the initial drafts (Oct 06) 
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Path Forward 


Implement remaining set of comments from Government 
Review Team (Nov 06) 


Government review team to comment on restructured and 
revised Guidebook content (Nov O06) 


Incorporate Government Review Team comments (Dec O06) 
Quality, Integrity Checks (J an 07) 
Public review (end of J an O07) 
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Summary 


e The Guidebook intends to provide information in three areas: 
— guidance on how to request, interpret, and use the results of appraisals 


— identify and evaluate strengths, weaknesses, and risks that may be 
associated with a particular offeror/team approach 


— gather and interpret information for effectively monitoring contractor 
processes 
e The Guidebook provides the basic information to apply CMMI-DEV on 
acquisition programs 
—- explanations of capability and maturity levels 
— differences between continuous and staged model representations 
- defines equivalent staging & high maturity/capability levels (levels 4 & 5) 
¢ The Guidebook designed for use by the Program Manager as a ready 
source of information and by the Program Management Staff by 
including detailed implementation guidance in referenced Appendices 
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Open Discussion 
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rmament Center <i 


Deliver war-winning technology, acquisition, test, sustainment 
...expeditionary capabilities to the warfighter 


Strengthening Systems 


Engineering Discipline 
to NDIA CMMI Conf 
15 Nov 2006 


Mr. lan Talbot 
u.S. AIR FORCE AAC/EN 


ian.taloot@eglin.af.mil 


DISTRIBUTION STATEMENT A: Approved for public release; distribution is unlimited. 


Integrity - Service - Excellence 


rr 7 7 
“4, 
LTS 
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Outline 


e Direction & Goals 
e Approach 


—- Process Focus 
—- Multiple Spirals 
e Spiral 1 
- Defining SE 
— Establishing a Baseline 
—- Results & Observations 


Path Ahead 

— Analysis & Mining 

- Coaching ... Mentoring | mprovement 
- Re-evaluating 


Final Thoughts 


061115 SEA to NDIA CMMI Conf; Talbot 3 


¢ Poor Systems 


Engineering | dentified 
as Factor in Acquisition 
Program Problems & 
System Failures 


Recognized Need to 
Revitalize SE 


Engineering Tasked to: 


— Perform a Center-wide 
SE Assessment by Oct 06 


—~ Where are we? 


- Baseline Enterprise 
Process I mprovement 


Direction & Goals 


lI mprove Program 
Performance & Reduce 
Technical Risk 


Ensure a Consistent 
Understanding of SE 


Ensure Core SE 
Processes are in Place 
and Being Practiced 


- Identify Opportunities for 
Continuous | mprovement 

- Clarify Roles and 
Responsibilities 

— Institutionalize “Best 
Practices” 
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Why Focus on Process 


e Process Discipline Leads To 
— Predictable Program Performance 
— Ability to Deliver on our Commitments 


Noti 


Optimized 


Quality 


Repeatable Defined Managed 


Process Discipline___ OO *=n 
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Ad Hoc 


Spiral Approach 4 
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Define Define 
Systems Best 
Engineering | Practices 


Develop 
Assessment 
Methods 


Assess 
Programs 
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Analyze 
Results 


Plan Actions 


for 
Improvement 


Spiral 1 


¢ CY06 


Define SE & 
Best Practices 


Develop Assessment 
Methods & Tools 


Train & Assess 
Programs 


Analyze Results 


| nitiate Actions for 
lI mprovement 


SE Models & Frameworks 


SDCCR MIL-Q MIL-STD-1679 
SCE——” STD- 


PSP 


People 
CMM DOD-STD 
IEEE Stds. 730,828 Nato \ 2268 — -2167A 
SA-CMM 829, 830,1012,1016 agapi49 
\ 1028,1058,1063 
EQA MIL-STBA~ 
FAA- " 498 — 
iCMM Trillium*— Baldrige Bs 
EIAIIEEE 
5750 
DO- ISO/IEC J-STD-016 
178B 
: 12207 
SE-CMM sae A 
/ SECAM / TickiT IEEE 
mere A \ AF IPD 1074 
CMM IEEE Guide = Q9000 AEEEIEIA 
a EIAIIS ISO 10011 12207 
MIL-STD__—» 632 ISO 15288 


-499B 
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Defining SE 


¢ Center CMMI° Acquisition e 9 Key Process 
‘ ‘ Module (CMMI-AM), 
Engi neering Version 1.0 Areas 
Steering eae ¢ 29 Goals 
Council oo e 117 Practices 
— Defined — e¢ 9 Generic 
iter Practices 
Criteria - 
- Approved [Best Practices ae ale 
Module & —— Ques ions 
[_mi-sr0-4088 |} a 
Approach ae ages 
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Key Process Areas 


e Technical Processes ee aaa 
—- Requirements & ‘<= 
— Design Sane 
— Verification/ Validation = 
— Transition = 
e Technical Management Processes 
— Planning 


— Risk Management 

—- Configuration Management 
— Decision Analysis 

— Technical Assessment 
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CMMI-AM Tailoring 4 


e What was tailored out 
— Elements not directly related to Systems 
Engineering (Contracting, | ntegrated 
Project Management, Logistics Support) 


— Section of Executive Questions that was 
Program Management and Enterprise 
Focused 

— Appendix addressing organizational 
process improvement 
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CMMI-AM Tailoring 


e What was tailored in 


—- Combined measurement and analysis and project monitoring and 
control and placed it in Technical Assessment 


— Strengthened reviews and audits in Technical Assessment 


- Combined requirements development and management in 
Requirements 


¢ Expanded to include elements from | SO-15288 
- Combined verification and validation 


- Developed Design consisting of Technical Solution, Architectures, 
Integration, Interface management 


¢ Drawn largely from CMMI -SE/SW/IPPD 
¢ Added “ilities” and Robust Engineering 
— Added DODAF architectures to Requirements and Design 


- Elevated Configuration Management from a generic practice and 
added some data management features 


—- Added Manufacturing and OSS&E to Transition 
— Reduced Generic Practices from 12 to 9 and reordered them 
— Added Qualification Questions for each Goal 
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Example 


Key Process Area: Requirements 


Goal: RG5 - Requirements are managed and controlled, and 
inconsistencies with technical plans and work products are 
identified. 


Practice: 


P1 Use a disciplined process for accepting, vetting, approving and 
providing requirements and changes to the developer through a single 
focal point. 


This process should prevent developers from receiving requirements 
changes from unauthorized sources that are outside the flow of the 
acquirer’s established configuration management process. Each change 
to a controlled requirement should be assessed for impact to the 
program’s performance, cost, and schedule baselines and to program 
risk. The existing cost, schedule, and performance baselines should be 
changed, as required, to accommodate the requirements change. 
“Requirements creep” must be avoided. A new requirement must be 
backed with money and vetted through a control process. 
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SEA Methodology 


Program does initial 
self assessment 


aeee Program SEA Review 2 hours 


~8 person hours 
~55 person hours P 


execution Self-Assessment Board leaderetap 


Co-chaired by Line EN Functional and 
Center Chief Systems Engineer, 


<o sila’ SEA Team attended by Program Leadership and 
pelea Su pport SEI Team Lead 


Scrubs answers reviews data provided 
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Enterprise View 


R-Requirements  V - Ver/Val P — Planning CM - Config Mgmt TA - Tech Assessment 


Legend D - Design T - Transition RM - Risk Mgmt DA - Decision Analysis 


Key Process Areas 

as of 18 Oct 06 
Program #1 | 
Program#2 [| TT 
Program #3 | 
Program #4 | 
Program#5 | 
Program #6 | 
Program #7 | | 
rl 
Program #9 | | 
Program #10 | | | aa 
ee ee 
Program #12 | | _ | 
— | | 
Program #14 | | | 
Program #15 | | == 
Program #16 | an] (ras 
Program #17 
Program#as | 
Program #19 __ =a | aS a 


Center Average 


High Bar Process Criteria Program Criteria Portfolio Criteria 
Scoring @ >90% of Practices @ >90% of Practices, No Red @ 95% Programs Green 
Criteria © 65-89% of Practices © 65-89% of Practices, NTE 1 Red © 75%-95% Programs Green, <10% Programs Red 
@ <65% of Practices @ <65% of Practices, 2 or More Red @ <75% Programs Green or >10% Programs Red 
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Process Area View 


Decision Analysis 

Planning YELLOW 
Requirements 

Technical Assessment 

Risk Management 

Verification & Validation 


Transition 


Design 


Configuration Management 


——_______—_ ayjag 


; Process Criteria 
Scoring @ >90% of Practices 
Criteria © 65-89% of Practices 

@ <65% of Practices 
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Evolving 


[_ 


In Plan 


a 


In Progress 


Sustaining 


Plan Actions 
for 
Improvement 
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see 


e Process 
—- Mostly Well Received 
- Training was a Time Saver 


- More Emphasis Needed 


¢ Production/Integration & 
Sustainment 


e Programs 
—- Few Documented Processes 
— Greater Benefit for Smaller 


—- Some Saddled with Difficult 
SE Strategies 
¢ 1067s, not COTS 


e Wants 
- Chief Engineers Handbook 
- Offeror Capability Reviews 
- Metrics for SE Health 


17 


Analyze Results 


e Systemic Analysis e Programmatic 
— Enterprise Wide Analysis 
Issues — Gaps ~ Program Specific 
¢ Understanding Practice Shortfalls 
os ~ Leadership from Line 
* Training Functionals 
— Specific Areas of ~ Functional 
strength Organization 
2 WAIDICk eat Provides Support 
Practices 


— Leadership from the 
Functional 
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Systemic Analysis 


L d R-Requirements V - Ver/Val P — Planning CM - Config Mgmt TA - Tech Assessment 
egen D- “ T - Transition RM - Risk ke DA - Decision eee 


Cc D> PR ee eet 

G94 WP 97 | 93 | 81 YF 2 YY 100 | om | 100 fT 71 | 
Weakness per] 97_—| Sst S| 2 co |e 8 
Strength 


ae eee) (pl ee) 
———————————————— 
ao ieee a ire 
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& Systemic Issues/Actions 


e Planning & SEPs 


—- Lack of Clear Definition for — Refine Local Process 
Technical Planning, Guidance, Tailoring, Add 
AF Policy Advancing Elements & Train 


e Requirements 


—- Lack of - Specific Shortfalls Treated 
¢ Comprehensive Use Cases through General Training & 
/ Mission Profiles Technical Planning Course 


e Requirements Validation 


e Life Cycle Stakeholder 
Involvement 


e Decision Analysis 
- Lack of Rigorous Process —- Define Process 
- Obtain Buy In 
Establish & Train Process 
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». Mined Best Practices 


te 


e Requirements Control & = ¢ Contract Incentives for 
Verification Working Reducing Cost and 
Group Increasing Reliability 


e |terative Requirements e Full Trust 
& Design Trade- off Integrated Teaming 
Working Group 


e Integrated & 
¢ Concurrent Engineering Overarching Risk 
to Ensure Successful Management Strategy 
Transition to Production 
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In Progress 


Implement 
Sustaining 


Improvement 
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Evolving 


Improving 


Define 
Guidance 
& Metrics 


Plan Actions 
for 
Improvement 


FYO7 


Continue Analysis of Results 
to Identify Gaps & Mine 
“How to” Best Practices 


Evolve Assessment Methods 


Implement Mechanisms & 
Framework for Coaching / 
Advising ... Programs 
e Share Results & Coordinate 
Improvement Efforts 
¢ Continue Development of 
Community of Practice 
e Initiate Development of 
Chief Engineers Handbook 
Implement Specific Action 
Plans to Address Areas of 
Weakness 


see 
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¢ Gaps 
- “Valley of Death” 


— Need to get to 
“Health” 


— AFMC Policy 


¢ Sustainment 
Emphasis 
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Evolve SEA Methods 


e Action Plans 


Mine Lessons 
Learned from 
Programs that 
Bridged the Valley 


Mine Willoughby 
Templates 


Map Shalls from New 
AF Policy 

Expand Module to 
“Health” 


AS 


Way 
>. 


Tool Expansion/Reorg 4 


e Technical Processes e Technical Processes 


Requirements = 
Design 


— Verification/Validation 
— Transition 

e Technical Management 
Processes 


Planning 
Risk Management - 


Configuration = 
Management _ 


Decision Analysis 
Technical Assessment 


Requirements 

Design 

Transition (to production) 
Verification/Validation 
Sustainment 


e Management Processes 


Planning 
Risk Management 


Configuration 
Management 


Decision Analysis 
Technical Assessment 
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Spiral 3 


Define 


Guidance 


Chief Engineer 
Handbook (How to) 


Evolving 


e eo r } ® 


In Plan 
EL] Direction 
In Progress Improving 
sy Plan Actions . . 
~ Implement far Chief Engineer 
esr Improvement Qualifications 
Improvement 
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Final Thoughts 


Credit to J eff Haars 


e Process Orientation e Self-Assessment 
— AAC History of Success Due — Easy Entry to Process 
to “Heroes” Approach 
Not Rigorous Process — Followed-up by External 
— Process Approach Will be Audit 
Paradigm Shift 
e Enterprise Process Group ¢ Investment Required 
— Process Approach Needs a - Vetting and Maintaining 
Process Group Documentation and 
~ Dedicated Group Compliance Measures... 
~ Process “Keepers” —- Audits, Auditors, Process 


— Mentors ... Coaches Group... 


- Education & Training 
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CMMI Opens the Gate: 
The Practical Relationship 
between CMMI and Stage- 
Gate® Decision Models 


NDIA CMMI Technology 
Conference & User Group 


Denver, CO 


AL ED ED 
FAEDID 
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Agenda 


ABB Overview 

Stage-Gate® Process 

= CMMI 

How Stage-Gate® and CMMI Complement Each Other 


= Summary 


AL ED ED 
FARDEID 
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ABB Overview 


= Leader in power and automation technologies 


= Enable utility and industry customers to improve 
performance while lowering environmental impact 


= The ABB Group of companies operates in more than 120 
countries and employs approximately 110,000 people 


= ABB became the first company in the world to sell 
100,000 robots 


= Avast majority of products at ABB have software and 
hardware components 


AL ED ED 
FARDED 
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ABB’s Organizational Structure 


Corporate Research 
ek oa _ - 


re 


Power Products | Power Systems 
== 


Automation Products ¥ 


Process Automation 
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ABB’s Products 


=m Power Products 


= Power Systems 
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ABB’s Products 


Automation Products 


Process Automation 


Robotics 


AL ED ED 
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Stage-Gate® Process Stage Gate 


Conceptual and operational roadmap for moving a new-product project 
from idea to launch. 


The Stage-Gate® process Is divided into a series of Stages (activities) and 
Gates (decision points). 


Stage| =m Stages consist of a set of cross-functional, parallel activities within a project. In 


Gate 


order to proceed from one stage to the next, these activities must be 
successfully completed and confirmed by passing a gate. 
=m Gates are passed during gate review meetings 


= Decision is made to either continue or stop a project based on whether a set of 
predefined Gate criteria have been met 


= Gates serve as quality control checkpoints 
= Gates decide the path forward for a project, including resource commitments 
= Gate decisions are based on 
= the set of deliverables from the preceding stage 
= aset of criteria/questions/metrics used to evaluate the project and its deliverables 
= Gate decisions: 
= Continue to the next stage 
= “Kill” the project 
= Hold or “freeze” the project 
= Recycle the project (re-scope and restart) Ah he iB 


FAIDID 
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Generic Stage-Gate® Process Stage Gate 


Discovery 


Idea Screen 


Sie (e| 


: Testing & 
Business Development 
Second Go To Go To WENICoF Vitel 
Screen Case Development Testing 


Go To 
Launch 


Post-Launch 
Review 
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Stage-Gate® Process Stage Gate 


= Conceptual and operational roadmap for moving a new- 
product project from idea to launch. 


= Provides a framework for better management of product 
development projects 


= Provides correct and relevant information for business 
decisions. 


= Ensures that the line organization is actively involved in 
the project and that the project status is made visible for 
all project stakeholders. 


= Based on the experiences, suggestions and 
observations of a large number of managers and firms in 
over 60 cases as observed by Robert Cooper. A. ED ED 
FAIDED 
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The ABB Gate Model 


= The ABB Gate Model has eight defined gates for 
controlling a project from a business point of view, each 
with a certain purpose as shown below. 


GO G1 G2 G3 G4 G5 G6 G7 


r TT 
—_— 
oO 
1s 
{e) 
pus 
ou 
— 
Ses 
oe) 
— 
” 


c —_— 
c c c o - oO 
2 fe} fe} } vn) 5 s 2 
‘c c=) =) - © = cS oO 
c = = 2) a e) a = 
=| Oo O =} ® a = Oo 
—_ o oF) xo) ad no = 
oO x x ° oO o oO 
~ Lu Lu =| n) > 
o ¢ S £ 2 £ 
a £ = ¢ 0 o 
” s § i= 
° oO 
O a 7) 
2. 
a 
S 2 
‘) 
re 4 
O 
2 A 2D ED 
a 
g PAID 
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ABB’s Gate Assessment Process 


Gate Assessor 


- Project Manager 
gh \ ] g 
\ 


Gate Checklist 


Gate Meeting Participants 


Cancel project 


I 7 
Gate 


5 Assessment to : : 
Gate . Report Gate ™ GO Continue project 
Assessment ” Meeting (with or without changes) 


™. Gate cannot 


be passed AL iB ale 
FAIDED 
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Benefits of the Stage-Gate® Process Stage Gate 


= Accelerates speed-to-market 

= Increases likelihood of product success 

= Introduces discipline into an ordinarily chaotic process 
=m Reduces re-work and other forms of waste 

= Improves focus via gates where poor projects are killed 
= Fosters promotion of innovative ideas 


= Achieves efficient and effective allocation of scarce 
resources 


= Ensures a complete process — no critical steps are 
omitted 


AL ED ED 
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CMMI Cnn 


= “Consists of best practices that address development and 
maintenance activities that cover the product lifecycle from 
conception through delivery and maintenance.” 


= Helps “organizations improve their development and maintenance 
processes for both products and services.” 


= CMMI can help: 
m set process improvement goals and priorities 
m provide guidance for quality processes 
m provide a yardstick for appraising current practices 
= Provide a framework for benchmarking among and across 
organizations 


Optimizing 
Managed 
Defined 
Repeatable 
Initial 


Capability Level 
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5 
4 
3 
2 
1 
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REQM PP PMC SAM MA PPQA CM 
Process Area 


. Baal 
Benefits of CMMI ae, 


= Increases the likelinood of meeting cost, schedule, and 
performance goals 


m Enhances collaboration with external stakeholders 


= Helps organizations define common, integrated 
processes for system development 


= Encourages proactive management techniques 


= Establishes disciplined, repeatable processes rather 
than chaotic, ad-hoc product development cycles 


AL ED ED 
FARDID 
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How CMMI and Stage-Gate® Complement Each Other 


= CMMI best practices support the activities occurring 
during the stages 


Idea Screen 


= Build © Testing & 

Business Development : 
Second Roce Go To GoTo (VE\UELiTela) 
Screen Development 


Testing 


Launch 


PPQA Post-Launch 
CAMA 


FPAIDED 
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How CMMI and Stage-Gate® Complement Each Other 


a Stage-Gate® provides a strong business decision and 
innovation focus not present in the CMMI 


Market Analysis 
Competitor Monitoring 


Business Case Development 


Intellectual Property Issues 


AL ED ED 
FARDID 


© ABB USCRC - 16 


How CMMI and Stage-Gate® Complement Each Other 


= CMMI provides more detail on best practices — especially within the 
Process Management and Support process area categories 


Stage-Gate models tend to be checklist based — 
focused on ensuring that criteria have been met 


CMMI provides guidance on specific practices that 
help organizations successfully achieve the criteria 


AL ED ED 
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Summary 


= Businesses combining the use of CMMI and Stage-Gate® 
= Products developed with repeatable, managed processes. 
= Products delivered on-time and within budget 
= Products meet market need 
= Products are profitable for the organization 


Products represent the highest level of innovation 


. Comal 
. + oo = 
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Contact Information 


Andy Cordes 

Software Architecture and Processes Manager 
ABB US Corporate Research Center 

940 Main Campus Drive 

Raleigh, NC 27606 


phone: (919) 856-3871 


email: andrew.cordes@us.abb.com 
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Background 


There are several straightforward 
ways to meet the practices of the 
Risk Management process area 


Beyond compliance, a Level 5 
organization asks “Can the 
effectiveness and efficiency of 
the Risk Management process be 
improved?” 


This presentation examines 
some of the techniques and 
opportunities being used by 
Northrop Grumman to build on 
the CMMI Risk Management 
practices 


NORTHROP GRUMMAN 
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A Quick Review of Risk Management 
Practices in the CMMI 


Risk Management - 
What’s Important to Improve? 


The Journey Ahead - More 
Opportunities for Process Improvement 


NORTHROP GRUMMAN 
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Review - Risk Management in the CMMI 


SG 1 Prepare for Risk Management 
Preparation for risk management is 
conducted. 


SP 1.1 Determine Risk Sources and 
Categories 
Determine risk sources and categories. 


SP 1.2 Define Risk Parameters 

Define the parameters used to analyze and 
categorize risks, and the parameters used 
to control the risk management effort. 


SP 1.3 Establish a Risk Management 
Strategy 

Establish and maintain the strategy to be 
used for risk management. 


SG 2 Identify and Analyze Risks 
Risks are identified and analyzed to 
determine their relative importance. 


SP 2.1 Identify Risks 
Identify and document the risks. 


SP 2.2 Evaluate, Categorize, and 
Prioritize Risks 

Evaluate and categorize each identified risk 
using the defined risk categories and 
parameters, and determine its relative 
priority. 


SG 3 Mitigate Risks 

Risks are handled and mitigated, where 
appropriate, to reduce adverse impacts on 
achieving objectives. 


SP 3.1 Develop Risk Mitigation Plans 
Develop a risk mitigation plan for the most 
important risks to the project as defined by 
the risk management strategy. 


SP 3.2 Implement Risk Mitigation Plans 
Monitor the status of each risk periodically 
and implement the risk mitigation plan as 
appropriate. 
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Process Compliance is Not Enough 


*Z = Acompliant process may not be efficient, nor effective, nor 
—7 sufficient to meet the customers’ needs 


_ 4 = Acompetitive environment requires a proactive approach to 
, improving all processes 


What do we expect to happen in a Level 5 organization? 


= Institutionalization 
" All projects implement Risk Management 


= A process improvement focus 
= Every practitioner is looking for ways to make the risk management 
process better 
" A data-driven culture 


= Data is gathered about the risk management process to identify 
ways it can be improved 


NORTHROP GRUMMAN 
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Risk Management is a Decision Management Tool 


Decisions With Respect to Level of Knowledge 


| Uncertainty Certainty 


(Probabilities) |More Some Certainty! “Perfect” Information) 


Uncertainty Uncertainty Near Certainty 
Choosing Selecting an Deciding on 
between architecture best COTS to 
emerging meet 

technologies requirements 


Many decisions fit somewhere 
in the middle where uncertainty 
can be “managed” 


NORTHROP GRUMMAN 
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Where Can We Look for I mprovements? 


Lean Six Sigma, applied to Risk Management 


= Voice of Customer 
= Who are the customers of the risk management process? 
." = What do they need? Desire? How do they judge success? 


= Voice of Process 


=" How efficient is our process? (e.g., risks identified/managed per 
resources spent) 


= How effective is our process? (e.g., identifying all risks) 
= Is there any unnecessary variation? 

= Are there any bottlenecks or time-wasters? 

= Can we reduce defects? (e.g., unidentified risks) 


= Voice of Business 
aa = What best-practices exist in the community? 
= What will our future needs be? 


NORTHROP GRUMMAN 
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SIPOC Process Map - 
Project-Level Risk Management 


[Suppliers [Inputs [Process | Outputs | Customers 


Determine risk Risk sources, | Risk identifiers 
sources and categories, (project 
categories parameters personnel) 


Historic risks Risk strategy 
Process parameters peep Project Risk 
Group Organizational Identified risks Manager 


guidance Determine risk Prioritized risks 
strategy 


External Areas of Risk mitigation Project 


Customer concern Identify risks plans personnel 


; Evaluate, ,; eT 
Project Areas of categorize, and Risk mitigation Project Risk 


personnel concern prioritize risks results Manager 


, Develop risk Lessons learned 
Manager experience 


Metrics 


Implement risk 
mitigation plans Adherence to Customer 


plans 
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Risk Management I mprovement Opportunities 
(1 of 2) 


Goal/Practices Improvement Opportunities 


SG 1 Prepare for Risk Management 
Preparation for risk management is conducted. 


SP 1.1 Determine Risk Sources and = More useful sources 
Categories = More useful categories 
Determine risk Sources and categories. 


SP 1.2 Define Risk Parameters = More insightful parameters 
Define the parameters used to analyze and 

categorize risks, and the parameters used to 

control the risk management effort. 


SP 1.3 Establish a Risk Management Strategy | = More effective strategies 
Establish and maintain the strategy to be used for | = Tailorable strategies 
risk management. 


NORTHROP GRUMMAN 


Copyright 2005 Northrop Grumman Corporation 


Risk Management I mprovement Opportunities 
(2 of 2) 


Goal/Practices Improvement Opportunities 


SG 2 Identify and Analyze Risks 
Risks are identified and analyzed to determine 
their relative importance. 


| | SP 2.1 Identify Risks = More efficient identification 
Identify and document the risks. = Few risks not identified 


SP 2.2 Evaluate, Categorize, and Prioritize = More efficient evaluation, 
Risks categorization, prioritization 
Evaluate and categorize each identified risk using | = Fewer risks mis-evaluated, 

the defined risk categories and parameters, and mis-categorized, mis-prioritized 
determine its relative priority. 


SG 3 Mitigate Risks 

Risks are handled and mitigated, where 
appropriate, to reduce adverse impacts on 
achieving objectives. 


SP 3.1 Develop Risk Mitigation Plans More efficient planning 
Develop a risk mitigation plan for the most More effecting planning 
important risks to the project as defined by the 

risk management strategy. 


SP 3.2 Implement Risk Mitigation Plans More efficient monitoring 
Monitor the status of each risk periodically and More effective monitoring 
implement the risk mitigation plan as appropriate. 
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Voice of Customer - 
Project Managers’ Top Concerns 


= More consistent risk identification, to reduce the number of 
unidentified risks 


= Better integration across teams and management levels 


= Better process tailoring to fit project size and risk level 


Experts Tools Policy 
| Metrics Process Guidance/ 
Training 


NORTHROP GRUMMAN 
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Risks Must be Rolled Up Throughout the Entire 
Stakeholder Pyramid Vertically 


| Risks are reassessed 
as they are rolled up 
to each level based on 
impact at that level 
NORTHROP GRUMMAN 
13 
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Better | ntegration Across Teams And 
Management Levels 


== 
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“Small Risks” Can Lead to Program Failure 


Requirements 


Acquisition 9’ Ge 
Process meee ¥ 


yr NN 

Design 
sd cole | ¥ (orm DY=\V-1(0) o)aatevali a. 

ar pment » 


> 


J 


J 


Test and Evaluation &. 
4 
Av, 


Operations ] 


Risky areas in our product 
realization processes 
provide opportunities for 
potential failure modes to 
be created and propagate 
causing failure 


NORTHROP GRUMMAN 
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Mission Success Requires Multiple Approaches 


s Risk Management Dashboards for Enterprise- 
re Wide Measurement 
Systems Engineering adele icclaa 

Effectiveness Communications & 


Independent Reviews Best-Practice Sharing 


Robust Governance Model 
(Policies, Processes, 
Procedures) 


Mission 
Assurance 


Training, Tools, & 
Templates 


Process (@) el=eelile)ats 


Effectiveness Effectiveness 


CMMI Level 5 for Software, 
Systems, and Services 


ISO 9001 and AS-9100 
Certification 


Six Sigma NORTHROP GRUMMAN 
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A Level 5 Organization is Never Satisfied.... 


A family 
of RM 
strategies 


Better 
metrics 


Opportunity 
Management 


More 
efficient 
statusing 


NORTHROP GRUMMAN 
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The Technology Strategy 
(@fo) nn) eke lan’ au 
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Imagine living an agile lifestyle with CMMI as your mother-in-law. 


What, with the constant pestering for improving yourself and providing better quality for her 
grandchildren. Not to mention always on your back about how much you spend on wasteful 
reyale (cre Ne) Ase 


Is she really a pest or does she just not understand you? 


Doesn't she see that you are improving yourself? That your quality does improve? That your 
cost variances and efficiency of resources are tighter than any of your friends? 
Or *her* friends’ children for that matter? 


“Oh! Look at how so-and-so's son-in-law bought his wife a big diamond! | bet *he* doesn't lead 
an agile lifestyle!” 


Yeah, so what! And exactly how does that big diamond contribute to improvement, quality or 
productivity? Huh? HUH?! 


So, you figure, if your CMMI mother-in-law is going to be looking over your shoulder all the time, 
you might as well show her how you are addressing improvements, quality, efficiency and 
productivity. It just might not be exactly as she's used to seeing it. 


So, in this session, “My Agile Life with CMMI”, we'll show you how Agile and CMMI can live 
itole(=iiatcyer-lalemel-\[amtarcm-\6)0)g@)\c-1me)mnvce)6) mu alera(e)ale(=1 me) Vicdecelcr- lal ale manrelialcleolamira\ ive 
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A What are we looking for? 

y NaCi-valcie-lijale moms at-lao1e mel ale(o)ecir-valellalen 
A Exercising the shared understanding. 


A The Take-Away. 
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A Few slides. 
A A few words on each slide. 
A Lots of discussion. 


A Session Purpose: 
seme [=val-ie-licme|t-l(ele|U(om-lp(eme(ct-) elo) eral e)e)acter- lle laure toes! 
Uhsto) ae] c0)0) ocean ccr-lelliccmlanle)icipalcvalrclilelap 
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What are we looking at? 
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Quick Poll ... 
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Your Mother-in-Law’s Way 


A How your M-i-L wants you to A How you want to live. 


IWiey Vv 
Vv Vv 
v Vv 
Vv Vv 
Vv Vv 
v Vv 
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TECHNOLOGY 
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Translating technology dollars into business sense.“ 


The Technology Strategy 
Companys™ 


>) ate | core Ol alelclacirelaellare 
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A Does your M-i-L have any of your experience to base 
her assumptions about your approach? 


A Does your M-i-L have any of your proof that what 
you're up to will work? 


Translating technology dollars into business sense.“ 
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Suppose... 


A Your Mother-in-Law wasn’t as fixated on path as 
gale (eiairstows) alsmlliskome) al asi-1 0) / &oe 


A Suppose you could demonstrate results and even 
account for the value in how you attainted them. 


b Aa (@)VVAAUOL0] (em nalismerar-lale(omlaliale psig 


v (this is where you come in again) 
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A How do we characterize “Traditional” and “Agile”? 
v (this is where you come in) 


A Traditional ) eNe Tits 
7 N y N 
7 N y N 
7X 7 
7X 7X 
7X A 
7X A 
y N y N 
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A Can we separate CMMI from development? 


Translating technology dollars into business sense.“ 
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Suppose... 


A CMMI was about process management, 
aleline (=17/-) (0) 0) pr -)¢1m 0) cele ty -1-e 


y Wa Neo} |(= more] e| (eo mr- (exere) 0a) mni@)an ele) sre (=1'/-) (6) 6)/ 71-101 me lel Vili otoe 


b Aa (@)VVAUOl0 (em calismerar-lale(omtaliale psig 


v (this is where you come in again) 
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/ NA ENV oma Comb) ale iolacsir-laleliaremce)mmere)anianlejal(er-lalaremme)me(-vadlale 
along with) your mother-in-law: 


bs Am ZolU i olelinm’\-lalmeatcmcr-lealomialialep 
s Ay Zo)0| ar-1 6) e)cer-(evalcromanlie lava elomeliiicieciale 


s AP No) dare)nii(cxe(e]l ale meat-lmy ele merelalicy.dicmelccomelliiclaciale 


v You can't always/only do things “her” way, or the way 
mo)ial-yeswme (emia) ale lon 


\ Am Lo) Um atc liom anlelacmlamece)anlanlelamiarciaalele 
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A What do we get at the intersection of Agile & CMMI? 
7 \ 


7 N 
7 N 
A 


Translating technology dollars into business sense.“ 


The Technology Strategy 
Companys™ 


Sl ayel cero MU late l<yesir-laleliale mars 


y Yan 100) 0X0)" (21 210 MOL \V(>1 (6) @)anl cial mccr= (Anise 


y Yay Noxore) 0) al mm celan av, -vavaiel ale marclmtclccts) 
time. 


A Manage the processes, not the 
elexe)e) {oy 


A Bureaucracy doesn’t work: 
v Are they incompetent? 
Vv Do they lack discipline? 
v Does Agile work w/either? 
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scrum Components 


Y Wau wa 0 l0 [0 (01m st-(er.dlelemr-lalemedt-lalaliare 
A Sprint Backlog and Planning 
A Resource Allocation 

A WBS 

A Daily Team Meetings 

A Peer Reviews and Inspection 


A Sprint Review 
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y Yam Halsm 6) cele [Uleim ey-(eL.a(ele mls 
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scrum master. 


PP 


CM 

RD 

iP 

Pl 

IPM 

RSKM 

DAR 

Cs Pa eal Van 4 ae | 


A Defines High Level 
Requirements and sets 
priorities. 


A Defines high level work 
break down structure. 


an an a a a a a a a 


y Weare Ware lei ilatom allele cavic 


release schedule. 
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demonstrable goals. This is 
usually at the use case level. 


A Tasks are broken down into hour- 
based estimates, anything over 16 
hours was broken down into 
smaller pieces. 


A The team creates tasks, estimates 
Valo meloitclaeallarcxomyiU sem ismele)iare mie mele 
what, everyone commits to the 
feasibility of the plan. 

v_ What can be done in 30 days with 
the resources we have at our 
disposal? 


an an a a a a a a a ae aa 


PP 


CM 

RD 

TS 

Pl 
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IPM 

RSKM 
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A Managed by the team, as 
aatc)oalelciacmere)aalpalimcemerciiilaremeate: 
Vela qrele)alcy 

A Members can play many roles 
at the same time: 

v Developer, Architect and DBA 
v Developer, Tester and 
Requirements Analyst 

y Nam (elpalelciar-lacmere)anlaaliiccremeomiatcs 
project and external noise is 
minimized. 

A The Scrum Master helps 
alleviate resource contention 
relate male)isioe 


PP 


MA 

CM 

RD 

TS 

Pl 
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RSKM 
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A A Sprint Goal satisfies a use 
case 


v “Allow a registered use to put 
itsiaatsm|aicemcalsi|mmsale)e)e)iale mere lai 


v “Allow a user to update the 


(ol UFeValdid(cxcmamatomcyale) 6) e)iale mer. lam 


A Each sprint goal is 
(ofslaalevarsiigsle)(sMm asl (staberele)ic 
ielarevite)ar-liiave 

v_ Show that this use case 
works, and has been tested 
and could be released as 
1 lateiie)ar-Uiay 


PP 


CM 

RD 

iP 

Pl 
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A Quick 15-30 Minute Stand up 
Meetings. 
A Answer 3 Questions: 
v What have you done since the last 
meeting ? 
\ Aen AVLat-lar-leomycelu me [6)| ale mcome (em el-iie)is 
our next meeting ? 
v_ What issues are you having that 
are impeding progress ? 
y Youn BY li Van als) ol=1e1dle)amr-lare MvAlsy1 6) Ava aice) 
team progress. 
A Daily Issues Management and 
Resolution. 
y Yann DY- li Vanna 00) [1m @xe)anlanr-laremelare 
@rolalige)MVVitaliamtaremsy-1 |i gare lareleliare| 
team. 
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PP 


NTAN 
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RD 
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Pl 
IPM 
RISK 
DANS 
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Peer Review and Inspection 


A Peer reviews keeps the team 
aalsanlelslesm ale)alcssie 


WPAN 
A Peer reviews are about 


natcvalce) diate mm arelm ele)icedialep CM 


RD 
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Pl 
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RSKM 
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7 an Oo) 90) 8) (e}omeral=xe1.d ele) alee lire, 
Xo) | fe r-tkct-wel(e)ale Mm igiom @)ae)(-xe1m cey-1e 
nate lomiat-lmer-lamelcmele)are 
iteratively and kept non- 
invasive. 
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Sprint Review 


A The Sprint review is a form of 
Ze lie li avaxeval=leL col micmelcitclanaliarcye, 
idarclmdalcm ale |e) @)celel0(e1m omelciiale 
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A Covers whether the product 
VVccksw 6] 6/11 malelalmelcrer-lUvoure| 
working version of the product 
sme] \Valalemr-MvdlcaiU sale mcomiate: 
le) cele lUlome) alcle 


A Product Owner (s) decides if 
jul ateidte)are liars laleme[ereliINarelas 
sufficient to be released 


a a ae a a a a a a a 


Translating technology dollars into business sense.” 


IN} BVA No )¥= 00) ol-1m40) 016) PAS) 


The Technology Strategy 
n sm 


Keys to Making " 
CMMI “Agile” 


A Reality-Based Process Descriptions and Life Cycles 


A Non-Sequential Process Flows 

A Non-Prescriptive Process Management 

A Getting “the right people on the bus.”* 

A Use “risk-avoidance-speak” not “model-speak” 


A Focus on Customer/Productivity/Profit/ROI 
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How We Made It Work (...) 


Specific 
Process 
Area | 
Process 7 Daily 
= Descriptions Management 
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Real Work Happens in —— 
Real Work Products 
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Templates 
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lf A=B and B=C... 


CMMI Happens Here: 


Templates 


Engineering 
Life Cycle 


Backlogs & 
Peer Reviews 
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the All-Important SCAMPI 


Process 
Implementation 
rors] ——* Indicator 
Descriptions 
come from here: 
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Conflicts & Synergies cou 


A Conflicts A Synergies 
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Conflicts & Synergies (me) 


A Conflicts A Synergies 
b Am c-lellilevarclmaveleiilaremis v Both want the SAME 
inherently non-Agile™, THING. 


can it be made to be agile? 


v Disciplined teams lead to 
Vv FAR are inherently neither successful implementation. 
Agile™ nor agile. 


b Ae lassveiielilelar-livzelilelai ls 
v Not all development teams natural. 
relacy (excl als elome|tsyel]6)i lacie p 


v Customer-focused isa 
v Tenacious persistence of (COTO) DR iaiiater 
paradigm perceptions. 


v Profit/ROI-focused is a 
GREAT thing! 
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A Narrowing-down each OU'’s reality takes work & time. 
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A People are the critical commonality... 
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Take-Away 
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A How each OU pursues that is unique and people- 
based. 


A |f CMMI sticks to managing processes and Agile 
incorporates practices as tasks, they get along. 


A Reality trumps authority and bureaucracy 
ete yeti 
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Use of SCAMPI*™ C for 
Agile Methodology 


Lockheed Martin Integrated Systems & 
Solutions (LM IS&S) 


Dorna Witkowski 
Mary Lynn Penn 


Approach to Agile at LM IS&S - 
Background 


LM IS&S CMMI® Maturity Level 5 
SE/SW/SS/IPPD 
Agile perceived as a way to do things faster, 
cheaper, better 
“We don’t need no stinking process.” 
LM IS&S starting to see RFPs “requiring” 
Agile and current Customers asking for Agile 


But there were different perceptions of Agile 
both within IS&S and by customers 


Approach to Agile at LM IS&S 


Therefore, IS&S saw the need: 


To provide a clear definition of Agile for IS&S 
and communicate that definition 


To understand the strengths and weaknesses 
of Agile and adopt it in the right way, for the 
right reasons, for our business 


To define the blend of Agile and plan-driven 
methodologies that will provide best value for 
Our programs and customers while still 
ensuring a disciplined approach 


IS&S Agility — It’s Not Just for 


Software 
‘ne 


not only 
Engineering 

but also Nes 
Management NST 
Procurement —- 


Human 
Resources 
= Program L land 
Business ae an 
Control Ee Assurance and 
— Development ae 


Ee Management 


IS&S Multi-Staged Effort 


Establish an Agile Core Team (ACT) 
Gather and share information and lessons learned about 
Agile across IS&S. Support proposals and customer 
“education” 
Generate an "Agile Reference Model" (ARM) which 
describes a Systems and Software Engineering Agile life 
cycle 
Use the ARM to identify and resolve Agile impacts to other 
organizations 
Define agile program/project selection criteria: 
Means to identify risks if Agile were to be used. 
| Validate the IS&S Agile life cycle through use on pilot 
programs. 
Communicate to the organization. 


Training materials, business development materials, 
engineering tool recommendations, etc. 


Use of SCAMPI*™ Cs for Intent 


Agile Reference Model (ARM) 
By February 2006, the Agile Core Team had 
established a draft Reference Model 
Focused on Engineering portions of process 
(Software and Systems) 

Organization wanted to ensure that the 

results of ARM were compatible with CMMI® 
Use of SCAMPI®™ C for /ntent was an ideal fit 
Plan to use a series of SCAMPIS™ Cs for intent 
as the Agile Reference Model gets refined for 
other process areas 


Agile SCAMPI*™ C Process and 
scope 


Used SCAMPI®™ Class C Method for Intent 
Limited scope to Specific Practices of Engineering Process 
Areas 
Obtained Senior Management sponsorship 
Worked with Agile Core Team to get access to 
documentation and identify interviewees 
Agile Reference Model was objective evidence for intent 
No implementation evidence available 
Used Practice Implementation Indicators (PIls) to record 
findings 
From documentation review and interview 


SCAMPIS™ C requires at least one item of objective evidence 
(direct, indirect, affirmation) for each practice 


Results reported to Agile Core Team and Sponsor 


Agile SCAMPI™ C Participants 


Experienced Team Members 


Name Introduction | Engineering | Management | Life Cycle Reporting 
to CMMI Experience Experience | Experience | Relationships 
(date (# years) (# years) (# years) 
attended) 
Team Lead October, 15 years 20 years Direct Report 
2003 to Sponsor 
Team Member 1 | February, 22 years 20 years Direct Report 
2005 to Sponsor 
Team Member 2 | October, 15 years 10 years Direct Report 
2003 to Sponsor 
Team Totals 52 years 15 years 50 years 


2 Interviewees from Agile Core Team 


About 1/3 of Agile Core Team 


Summary of Results 


Very high correlation of intent of ARM to CMMI® Specific 
Engineering Process Area 


41 “Green” Specific Practices 
Intent adequately addressed 
4 “Yellow” 


Intent partially addressed (RD SP3.1, TS SP3.2, VER 
SP2.1, VER SP2.2) 


1 “Red” 

Intent absent or poorly addressed (VER SP2.3) 
Recommended that documentation be enhanced to make 
Sure intent is explicit 

Document how the method handles interfaces, reviews, 
recording decisions, and constraints 
Recommended method be updated to include: 


Measurements, Operations Concepts, End User 
Documentation, Peer Reviews 


Actions Since SCAMPI™ C 


Measurement section generated for ARM and 
is in the review process 
“Product Vision” defined in ARM Is consistent 
with Concept of Operations 
Pair Programming being viewed as one form 
of “Peer Reviews” 

Other reviews being defined in ARM 


Next SCAMPIS" C for Intent being planned 1Q07 
10 


Summary 


Agile Reference Model (ARM) being defined by IS&S 
to ensure consistent, best-value approach 


Customers are driving the need to address 


Organization is driving the need to maintain CMMI 
compliance 

¢ Goal is to define the blend of Agile and plan-driven 
methodologies that will provide best value for our 
programs and customers while still ensuring a 
disciplined approach 

¢ SCAMPIS™ Cs for Intent are an effective tool to 
ensure ARM is consistent and compatible with CMMI 
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Organizational Synthesis - 
CMMle, The Glue That Binds 


Synthesizing: 
= Business Process Model, 
= ISO Standards, 
" Life Cycle Management, 
= Program Direction and 
= CMMle 
into a Business Process Management System 


Eileen Arnold, Patrick Waddick, Helen Addie, Christine Hulse and the ASD 
Engineering Team 
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ASD’ S 
Concurrent Initiatives 
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BAE Systems Concurrent Initiatives 


The major concurrent initiatives and 

business drivers used for illustration of our 
organizational system synthesis as presented here 
include: 


= ASD’s Business Process Model 

= |ISO Standards 

= BAE Systems Operational Framework and 
Life Cycle Management Process 

= CMMI® 
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Change Is Inevitable 


The pace at which change transpires and 
the complexity of that change is dependent 
upon the: 


1) Number of concurrent change drivers 
2) Degree to which the drivers are instituted 
3) Cultural acceptance of the drivers 


4) Capacity (Resources) to implement the 
change, and 


5) Application of system synthesis 
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Our Business Approach 


Compliant with ISO; Utilizes CMMle Best 
Practices; Leverages Assets (PEOPLE, TOOLS) 


Goal EBIT Growth Productivity 


Customer 
Satisfaction 


Focus iiiiye)ceeiilcalae rom AW) 


Strength aig wadale Be to [UComm =>.) 
Methods > 


Foundation 3) xe QESC 


Continual 
Process Improvement 


Knowledgeable Employees (KM) 


Using Effective Tools, Processes 


Dedicated Leadership (at all levels) 
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ASD Process Improvement & History Milestones 


ISO 9001:1994 BPM Project = 1SO 9001 & swcmm, '$0 14001 scm 
Certified Began 14001 Integrated sae Certified Level 5 
BPM Integrated) SW CMM (Planned) 
SW With ISO Level'd SWISE CMMI 
CMM L2 ML 3 
SW 
CMM L3 


1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 


Business and Product Development Process Improvement 
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ASD’S 
Business Process Model 
Initiative 
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ASD’s Business Process Model Architecture copyrignt 2001 Bae systems 
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Manage Process Performance 
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Manage Human ers Manage Financial Manage Physical Manage Information 
Manage Manage Facilities 
Resources Resources Resources Systems 
Program & 
Project 
Performance OT CM 


Develop 


Manage Business 


Contract 


Current  OPF 
-  OPD 
_— synthesis oip 


Tech. : 
focus 


Initiate 
Internal 
Projects 


sjonpold 


> 
° 
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=|] Manage 
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MA Product 
Manage 

Measurement 
and Analysis 


Activity 


BPM - CMMI® Process Area Mapping 


PPQA 


Manage Process Performance 


Conduct Execution Manage Regulatory Manage Quality and Take Corrective, Preventive, 
Excellence Review Compliance Environmental Systems & Improvement Actions 


Once printed, the hard copy of the on-line document may not be B, ~ very. curregi revision on-line. 
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ISO Standards 
Initiatives 
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BAE SYSTEMS 


ISO 9001 and ISO 14001 


ISO 9001 standard emphasized 


Defining sequence and interaction of processes 


Customer requirements and customer 
satisfaction, for both external and internal 
customers 


Process measurement and/or monitoring; data- 
driven decision making 


Continual improvement 


Increased emphasis on management 
responsibility 
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| _ ISO 9001: Implied Process Hierarchy —__ 


Fits Well With 
ASD’s BPM 
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CMMI® and ISO are looked upon as 
people and process enablers to 
accomplish performance sharing, lean 
initiatives, safety, and other critical 
business support initiatives 
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BAE Systems 
Operational Framework (OF) 
ife Cycle Management (LCM) 


Business Process 
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Acquisition by BAE Systems 


BAE Systems Operational Framework (OF) 
(governance framework for all Business Units) 
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Life Cycle Management (LCM) 


LCM - The Framework for the Management of 
Projects in BAE SYSTEMS 
Phase - Contract - Design - Bid re 


LCM Supporting Material and Best Practice 


Pees Engineering Commercial Procurement prune 
a 
MANDATED ~ 
BY CORPORATE Ww 

BEST PRACTICES 

AND LESSONS PROGRAMS rye GANDATED “ 
LEARNED PROCESSES BY PROGRAMS w 
a 
APPLICATION OF PROCESSES AND IMPLEMENTED 4 
PROCEDURES TO INDIVIDUAL PROJECTS ON PROGRAMS Fa 
x | 
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Another Synthesis Opportunity - OF 


1) All employees have an awareness of the 
Operational Framework 


2) Local policies and processes are in place to 
implement the requirements of the Operational 
Framework and all mandated policies and 
processes 

3) Appropriate training and monitoring processes 
are put in place to ensure proper 
implementation of the Operational Framework 
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Synthesizing Activities 
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ASD" s Process Based Manac ement Vision 


Process Maps Procedures 


e usiness 


aeenng Operations 


Continuous 


EER Review 
Improvement 
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SEIl’s IDEAL™ Approach to Process 
Improvement 


Learning 


Analyze 


and 
Propose : 
Future WENT) 
Actions Implement 
Solution 


Refine 
Solution 
Stimulus for Change met Build Charter 
9 Context Sponsorship Infrastructure 
Pilot/Test 
Solution 


Initiating 
Characterize 
Current & 
Desired States 


Create 
Solution 


Diagnosing Weagy: 


Recommendations 


Plan 
Actions 
Set 
Priorities Develop 
Approach 


Establishing 


BAE Systems 2006 All Rights Reserved 0 of 31 


NDIA Presentation November 15, 2006: ASD Engineering 


Legacy Documentation (Stove Pipe Initiatives) 


Business Process Model 


PO] 


ISO 2000 
Policies 


Organization 


Standard 
And Processes 


Processes 


ISO — HW Oriented 17 CMM — SW Oriented 
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General Approach For Synthesis 
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Why Focus on Processes? 


PEOPLE 


TECHNOLOGY 


PROCESS 


Major determinants of product 
cost, schedule, and quality 


BAE Systems 2006 All Rights Reserved 72 of 31 


NDIA Presentation November 15, 2006: ASD Engineering 


Process Institutionalization 


Process institutionalization has been achieved when: 
= A policy relating to the process exists 
The process is defined and documented 
Training exists and has been delivered 
Required resources are allocated 


The majority of the process is or has been exercised frequently (as 
appropriate to the project lifecycle) 


The execution of the process is measured, managed, and verified, 
and corrective actions are taken 


It is clear that the staff understand their processes, can discuss 
related artifacts, and believe in the effects 


Institutionalization: The ingrained way of doing business that an 
organization follows routinely as part of its corporate culture. 


Source CMMI Version 1.2 


BAE Systems 2006 All Rights Reserved A of 31 


NDIA Presentation November 15, 2006: ASD Engineering 


Process Influence on EBIT 
ISO 14001 
BPM Project ISO 9001 & Certified 
ISO 9001 :1994 Began 14001 Integrated CMMI 
Certified Level5 
: ? SEISW CMMI 
3PM Integrate aur aK ee (Planned) 


With ISO 


Level 4 SW CMMI 
ML5 


ee 
CMMI Displayed Advantages: os af 
Optimizes entire system not just parts 
Process skills easily transferred 


Provides consistency, eases skill transfer, efficient ops 
Resolved system-software integration ae i a 
ees ee ee | 
f Performance 


2000 2001 2002 2003 §$2004 #«2005 


2006 2007 #2008 


1999 


1998 


1996 1997 


Business and Product Development Process Improvement 
Correlates with financial performance : 
of 31 
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Business Advantages of Organizational 


Synthesis 


= More efficient use of resources In 
developing, documenting, deploying, 
maintaining, and knowledge management 

= Consistency in understanding, less 
confusion — unified process alignment 

= Improved communication due to common 
language, processes 
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Summary 
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In Summary 


1) Synthesis of the Organizational System 
presents an opportunity to operate in an 
effective business manner, improving 
coordination by reducing redundancy 

2) For ASD, CMMle and the need for 
institutionalization provided the glue to bind the 
organizational initiatives into a synthesized 
operation 


3) Change happens — embrace the opportunity 
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Key Enabler — CMM Practices 


Customer 
ISO Influence CMM 
9000 CMM 
9001 PCMM 
14000 see — 
Business CMM-X 
Process 
* Organization Model 


¢ Culture 

¢ Governance 

¢ Core Business 
Processes 


¢ Organization 
¢ Culture 
¢ Business Objectives 


OF Laws and ASD 
Regulations 
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The Glue Results In ... 


Customer 
ISO Influence CMM 


a 


ASD 
Business 
Process 
Model 


s 


Laws and 
Regulations 


Synthesis of Stronger Increased 
Initiatives Business Process Model Success of 
OF Standard Processes Business Goals 
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Organizational synthesis is a journey, not 
a destination. 


Integrating the organizational system is 
an evolutionary activity even if the 
company maintains its current 
initiatives, driven by the business need 
to change and improve. 
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Operations and Maintenance — Not Just a Scaling Issue 
Laurence Gill 
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Operations and Maintenance Environment 


Challenges Facing O&M Projects - 1 


¢ Supported applications are already developed and fielded 
Real time environment 
— Releases are typically added to a live system 
@ Typically part of multi-contractor team 
— May need to acquire some of the original developers 
@ 24-7 operation (actual or virtual) 
« Responsible for overall system performance via Service Level 
Agreements 
@ High variability in workload 
— Help Desk Functions 
— Preventative Maintenance 
— Testing 
— Development 
— Release Management 


Challenges Facing O&M Projects - 2 


@ There are few, if any, recognizable development activities that the 
organization controls that map to all of the engineering practices in the 
CMMI model 

Customer owns process 

— Contractors may be able to influence the process 
— Customer controls changes to the process 
— Customer owns tools that support the process 
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Breakthroughs 


@ Classic engineering is not a prime focus 
— Tier 1 help desk tickets are often administrative or training 
— System problems get fixed immediately 
— Content of releases is less critical than implementation into live environment 
Most O&M contracts are Level of Effort (LOE) 
— Changes to the customer system operate differently 
— Work force is pre-determined; requirements are based on size of team 
— Schedule needs to adjust to the world of the application or network 
@ Many processes are performed at two levels 
— Project management exists for the O&M contract and the specific customer 
application 
— Support processes support the contract artifacts and the application 
environment 


Two projects exist — ours and the one we perform for the customer 
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Framework for Implementing CMMI 


Process Control 
Contractor 


Recruiting Planning & Forecasting 


Program Execution 
through: 
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Skilled Resource 
Utilization (SRU) 
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Framework for Implementing CMMI - Below the Line 


Operations and Maintenance activities occur below the line 

« Work performed at customer direction performs similar 
processes but for different purposes 

¢ Validation determines whether the work we perform below the 
line meets customer expectations 
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Framework for Implementing CMMI - Above the Line 


Process Control 


Contractor Program Execution 
through: 


Recruiting Planning & Forecasting 
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Skilled Resource 
Utilization (SRU) 


GHuuaeulbuz 


v 
as 

2. 
© 
° 
ot 
< 
© 
S 
© 

Ke! 
© 
= 
Oo) 
=| 
ot 


Sa8ssad0ig Woddns 
juUowebeue|\| SS8D0I1d 


¢ The products being provided are resources capable of performing the 
work below the line 


¢ All of the CMMI Process Areas are applicable above the line 


— People are products; Systems are the resources we provide (staff 
and supporting non-staff) 


— Verification is against the requirements of the contract — not the 
verification activities performed for the customer 


@ Processes above the lines are under our control 
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Above the Line Processes 


SRU Process Process Description 
Define the Work (DTW) Identify customer needs and expectations and 
define project requirements. 


Design the Staffing (DTS) | Develop a resource solution that will address the 
requirements of the contract and the expectations 
of the customer — with an understanding of a real- 


time variable-workload environment using LOE 
resources. 


Define Project Approach | Develop plans and processes to be used to 

(DPA) manage the resources provided to the customer. 

Fill the Position (FTP) Acquire the staff and non-staff supporting 
resources to perform the services. 


Monitor the Project (MP) | Manage the requirements of the project. 


Implications to the Business 


¢ Create breadth versus depth in staffing solution — 
virtual workforce 
@ Have to know best practices for below the line — 
influence 
¢ Leading indicators to performance 
— Key performance indicators for staffing 
— Cannot ignore leading indicators for SLAs 
¢ Develop multiple alternatives to provide services 
and select the one that best addresses the contract 
and environment 
@ Understanding what is required below the line is 
key to success above the line 
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Summary 


With our business-tailored interpretation of the CMMI we were able to 
apply best practices to the entire project, not just small bundled 
releases 
We were able to identify the O&M environmental challenges and 
develop a business solution that can be applied to our entire enterprise 
— Organizational standardization of processes 
— Project’s defined process 
By separating the tasks we control from those that we don’t, we were 
able to develop a better framework for implementing the best practices 
using the CMMI 
The processes developed mirror our business 
For more information about the implementation of the engineering 
process areas, join Kathy King tomorrow for the Engineering in a Non- 
Development Environment presentation 


BAE SYSTEMS 


Contact Information 


« Laurence Gill — BAE Systems Information Technology 


—larry.gill@baesystems.com 


Thomas Zience - BAE Systems Information Technology 


— thomas.zience@baesystems.com 


Customer Success Is Our Mission 


When OID Crosses Time Zones, 
Disciplines and Customer Base 


Network Centric Systems 


Engineering icone 
Laurie Haack Ex 
. NCS CA 
Michele Wall Eo 
NCS TX 
NCS FL 


EPG OID 


EPG OID 


November 15, 2006 
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Introduction 


» NCS has transformed from five sites with separate sets of 
processes to a common set of processes shared across NCS by 
all engineering disciplines. 

» As part of that transformation, regionally independent 
improvements activities were replaced with an NCS-wide 
organizational level improvement activity. 


a» This has enormous implications for the ‘org’ in Organizational 
Innovation and Deployment. 


=» This presentation will 
— present the components of the NCS organizational improvement 
infrastructure 
— discuss how the organization improvement transformation occurred 


— Include a summary of key lessons learned 
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>. Overview 
Solution 
Journey 
Lessons Learned 


Customer Success Is Our Mission 


Raytheon 
Network Centric Systems Network Centric Systems 


NCS develops and produces mission solutions 
for networking, command and control, situational 
awareness and air traffic management 


Major programs include civilian applications, 
command and control systems, integrated 
communications systems and netted sensor 
systems as well as components to create these 
systems 


President: Colin Schottlaender 
Headquarters: McKinney, Texas 


Five Primary Engineering Locations with 4000 Engineers: 
Marlborough, MA 
Ft Wayne, IN 
St. Petersburg, FL 
Mc Kinney, TX 
Fullerton, CA 


CMMI Maturity Levels 
Starting point — Three level 5s, Two level 3s 
Goal — NCS-wide level 5 for SW, SE and HW 11/16/2006 | Page 5 
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Why Take on the Big Org Challenge 


= General business = The big org has so 
direction to have one many businesses, 
customer voice and ppAA Ps products, and 
to execute % customers — 


prioritization is hard 


= Team building cross 
region and cross 


seamlessly across 
regions 


= Invest $ where the 


overall impact is discipline is 
greatest complicated 

=" Would enable CMMI = Communication 
LS5 OID infrastructure 


_ , , demands are greater 
= There was minimal sharing of improvements 


assets or improvement project results across = The big org doesn't 
sites. have a personal face 
, , and a single authority 
= Redundancies existed — several sites doing below the executive 
improvements projects on similar topics. level 


= Limited resources were split across the sites. 


Each business must do their own analysis of benefit. 


11/16/2006 | Page 6 
Copyright © 2006 Raytheon Company. All rights reserved. 


Overview 


va? SOlution 


Journey 
Lessons Learned 
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NCS Organization Improvement Process 


Collect Analyze Select Execute Monitor Learn 
Proposal Proposal Project Project Project Project 
Advisory Advisory 
Submit Evaluate Evaluate Execute Execute All 
Screen Rank Complete 
IPPDB IPPDB IPPDB IPPDB iMetrics Various 
R60 DB 


Engineers Council Engineers EPST 
OIC EPST Council 


Council NCS ETQ LT Metrics Council 
EPST 


24x7 Monthly Annually 24x7 Monthly 24x7 
& Weekly & Quarterly 


Six Step Process that leverages R6o and Existing Org Structure 


La Lyurcuvy | aah elated 
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NCS Organization Improvement Process 
Guidance & Support 


Work Instructions 
Enablers 

Tools 

People 
Communication 
R60 


Enablers 


CPA Bulletin Template 

CPA Process Senior Management Review Template 
mprovement Project Plan and Report Template 
Organizational QM Plan Template 

iPlan Import Justin-Time Training 

mprovement Project Quad Chart Template 

ETQ IP EVMS Enabler 

ETQ Focus Area EVMS Consolidation Tool Enabler 
ETQ Focus Area Status Report Enabler 

ETQ IP Inchstones for Behavior Development Enabler 
ETQIP Inchstones for Pilot Project Enabler 

ETQ IP Gate4 or Gate5 Enabler 

ETQIP Gate5 Checklist Enabler 

ETQ Focus Area Project Status Report Enabler 

ETQ IP ROI Tool Enabler 

Pilot Plan and Results Report Enabler 

Pilot Plan Checklist Enabler 


* 
* 
* 
* 
* 
* 
* 
* 
* 
* 
* 
* 
* 
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Raytheon NCS Common Process Architecture 


Customer Success Is Our Mission Raytheon Home | Directory | Search | Newsroom | Collaboration | Help 
Common Process Engineering Process Behavior 
Architecture Councils Assets Rollouts 


NCS Policy for CPA 
CPA Contact List 
Bulletins 


Organizational Processes 


Release Notices 
Dashboard 
Process Viewer 


Communication Briefings 
Process Viewer Overview * Improvement Process Comm Brief 

* Evaluate Organization Improvement Proposal 
* Submit Organization Improvement Proposal 


Events 
News Archive 
Change Request (CR) 


Improvement Proposal 
(IP) 


CR Reports 
ConOps 


Plans 


Overview 


NCS CPA Work Product Management Plan 
NCS CPA Work Product List 


NCS Engineering Organizational Quantitative Management Plan 
NCS CPA Stakeholder Involvement Plan 


Tools 
NCS Learning 


NCS Process * 


Organization 
FAQ 
Glossary 


NCS ETQ Page Work Instructions 


Related Sites 

Help Desk 

CMMI 

IPDS 

R6S 

Fort Wayne, IN 

St. Petersburg, FL 
Marlborough, M4 (NE) 
Fullerton, C4 


* Organizational Improvement Process VVork Instruction 
CPA Bulletin Work Instruction 

Appraisal Planning and Execution Work Instruction 
CPA Management Review VVork Instruction 

CPA Process Deployment VVork Instruction 

CPA Process Improvement Planning Work Instruction 
Execute Improvement Project Work Instruction 


* 


IP Infrastructure includes direction, people, tooling and enablers. 
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Work Instructions 


Collect Analyze Select 


4Y http: //ncscpa.ncs.ray.com/downloads/OP_PRM_MD_Organizational_Improvement_Process_WI_doc - Microsoft Internet Explorer 


File Edit View Format Tools Table GoTo Favorites Help 


eek - > - OE | Qseach Facies Gre OH So ay 


Raytheon 


Network Centric Systems 


3 http://nescpa.ncs.ray.com/downloads/OP_PRM_MD_Execute_Improvement_Project_WI.doc - Microsoft Internet Explorer 
File Edit View Favorites Tools Help 


Back ~ > - @ [2] | QSearch GyFavortes BPMedia <H| FI Eby at S| 


Address |4] http://nescpa.nes.ray.com/downloads/OP_PRM_MD_Organizational_Improvement_Process_W.doc 


Address 


i] http://nescpa.nes.ray.com/downloads/OP_PRM_MD_Execute_Improvement_Project_Wl.doc 
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Work Instruction Number | U0141DFP 
Revision | A 
Organization Improvement Process Supersedes | - 
Date | 9/11/2006 


The purpose of the Organization Improvement Process is to establish a process for 
collecting, selecting, and executing organization improvements for the NCS 
Engineering, Technology and Quality organization and for the regional engineering 
organizations within NCS ETQ. Concepts fram this work instruction may be applied 
at the program level 


= 
Introduction 


Improvernent Proposals 
Improvement Projects 
Organization quality measurements 


Input(s) 


Organization performance measurements 
Organization business goals 


Requirements Responsible Role 

i) Appoint Organization Improvement Process Lead | NCS ETQ Vice 
President, Regional 
Engineering 
Directors 


Provide Resources for Ornanizatinn Imornvement | Ornanizatinn 


. a : Ce ae 
Raytheon 


Network Centric Systems 


__ {EERE A EE ERE CEE 


Nurnber | U0141DFK 
Revision | A 
Supercedes | - 
Date | 9/11/2006 


Work Instruction 


| Execute Improvement Project 


The purpose of the Execute Improvement Project Work Instruction is to describe the 
process for planning, conducting, and evaluating an improvement project. There are 
four different stages considered when an improvement project is planned: Research, 
Develop, Pilot, and Deploy. An improvement project may include one or more of these 
stages. The improvement project plan, activities, results, evaluation and 
recommendations are documented and shared with the organization. Refer to 
Appendix A, Figure 1 for a graphic presentation of an Improvement Project. 


Improvement Proposals 
Approved Improvement Project 
Step 


Identify Improvement Project Purpose, Funding, 
Sponsor, Review Board and SME 


| 2 | Establish a Raytheon Six Sigma (RBS) Project 
[ 3 | Determine Applicability of a Research Stage 
| 4 | Determine Applicability of a Development Stage 


Introduction 


Input(s) 


Requirements Responsible Role 


Lead 


| Lead 
| Lead 
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Tools (1 of 3) : Network cae Systems 
Improvement Proposal and Project Database 


File Edt View Favorites Tools Help | «2 | 
P > - @ 2) | ASearch (Sj Favorites EI'Media <4 E- (4 Sw - 


Address 7] @Go | Links ” 


Organizational Improvement Help with Submit | 
Submit Proposal/Advisory 


Type of Improvement Proposal (IP)?: « [Proposal x] 
If Advisory, how will it be funded?: [Select Below 7] 
Source of the IP idea? + [Select Below x] 


What Disciplines does the IP fall under? + HYV - Hardware Engineering 
Check all that apply. Choose multiple disciplines by using PE - Prograrn Engineering 


the "Shift" or "Ctrl" buttons. QE - Quality Engineering 


SW - Software Engineering 
SE - Systems Engineering | 


Short Title for the IP: « 


Description of the IP: « a 
Describe the proposed idea. The description must be less 

than 8000 characters. If you believe the description will 

be longer than this, please attach the description in 

document form. 


“4 NCS Common Process Architecture - Microsoft Internet Explores 
“File Edt View Favorites Tools Help 

sa Back - = © QD [2] G | GSeach |] Favorites 
Ades 


Raytheon 


Customer Success Is Our Mission 


Common Process 
Architecture 


NCS Policy for CPA NGS Cammon Process Architecture 
CPA Contact List 
Bulletins 


The Common Process Architecture (CPA) is an 
infrastructure for a sustainable, affordable, no 
redundant, integrated Engineering processes, 
integrates Hardware Engineering, Quality 
Engineering, Program Engineering, Software 
Engineering, and Systems Engineering. It is 
Capable of being extended to non-engineering 
processes such as Program Management and 
Supply Chain Management. The CPA also maps 
PDS, 459100, and the Capability Maturity Moi 
Integration (CMMI). 


Release Notices 
Dashboard 
Process Viewer 


Process Viewer 


Events 


News Archiv 


nge Request (CR 


Improvement Proposal 
(IP) 


I | 
Attach a file (Optional): Browse... | 


Type in the file path or click BROWSE... and select the file. 


ConOps 
Overview 


Why is a Common Process Architecture desiral 


Tools 


; . Facilitate deployable work packages acri] File Description: * Zz 
NCS Learning » Facilitate multi-site project execution Required ONLY if attaching afile. 
NCS Process + Optimize deployment of improvements in] Enter a short description, less than 100 characters. ira | 
Organization » Provide effective sharing of best practic 
FAQ . Minimize redundant process efforts 


Please click SUBMIT just one time. The file upload may take a moment. 
SUBMIT | CLEAR FORM | 
* THE ASTERISKS INDICATE FIELDS THAT ARE REQUIRED ha 


CC pie 


Glossary . Decrease process sustaining casts 


é ] 2 Hierarchical Menu Trees Created 


Link is easily Available Proposal Submission Form is Simple 
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Tools (2 of 3) 
IPPDB MetaData Available to All 


y NCS Organizational Improvement - Microsoft Internet Explorer -(O} x] 


File Edit View Favorites 


Help 


Tools 


ye 


Bak ~ > ~ @ GI) QSearch fjFavortes GMedia <4) GS~ [A | ~ = 


Address 


Raytheon 


x] @Go | Links 


i loecmelgerlalreicelercimiteehasiitaals 


Customer Success Is Our Mission Directory | Search | Newsraom | Collaboration | Help 


Submit Proposal | Browse Proposals | Submit Project Browse Projects | Metrics | Administration 


NCS Org Improvement 
6. Add Architecture C # 


9, Ontology Integratio 
. Loading Analysis 
. Test Driven Devele 


. Consistency of be 
. Inconsistencies ir 


7. Tech Focus Area: | 
8. Shared repository f 


. test_of new versior 


. Small program tai 


. Composite Progre 
. Standard Tool Sel 


. Define Process fo 
. Task Work Stater 
. Shared NCS ETQ 
. Centralized mana 
. Comparision of bu 
. Collaborative Sup 
. Technical Require 
. Test Excellence 

. Software/Systems 
. MS2&A: Infrastruct 


. Architecture Spec 
. RCA Trained Arck 


. Architecture Proc » 
rf 


Filter | Reset Filters | Status Page Show Mine | 


ID Prop/Adv Title Status Review Board SME Score 
i I [au x] [au | [au x] [ 


Combinatorial Design 


Methods for Testing Evaluate 


De Wolf, LeifR 


Adopt Model Based 


Architecture Assign 


AUTOMATE CODE 
DEVELOPMENT REVIEVY 
PACKAGES 
Add Architecture Quality 
cherks to Insnerction Criteria in 

Spawn Proposal 


Closed 


Evaluate Venkatesan Sanieev 


Tab Instructions 


Evaluate | Assign | Hold | History | 


I Investigate x] 


Submit | Screen 


Sub-Status: Evaluate Attachments: 


Upload A File 


SME: Delete A File 


[Venkatesan, Sanjeev pa 
Update Score | 


44 


Current Files: 


©) Evaluator analysis. 

ie Shows the orncess of ATAM faa 
Evaluate Comments: 

The SWC reviewed this on 3/14/06. It is assigned to 

Sanjeev as the Architecture SWC point of contact: Bring 

the checklist concept (might need updating) to Growth 

Architecture team. >| 


ié] 2 Hierarchical Menu Trees Created 


Proposal and Project Frames list all content, 


Local intranet 


support 


filtering of summary information and allow access to 
detailed information about a proposal 


S| NCS Organizational Improvement - Microsoft Internet Explorer 

File Edit 
Back > = ~ fat | QsSeach Gy Favorites EI\Media 
Address 


View Favorites Tools Help 


Customer Success Is Our Mission Directory | Search | Newsraom | Collaboration | Help 
NCS Org Improvement | Submit Proposal | Browse Proposals 


Organizational Improvement 


Organizational Improvement Metrics 


Proposals 
Here you will find different metrics reports showing the status ofthe Orga 


Currently in each State 
als Status Table 


Here you will find different metrics reports showing the status ofthe Orga 


Projects Currently in each State by Month 
Projects putinto each State by Month 


Projects Currently in each State 
Projects Status Table 


ié] 2 Hierarchical Menu Trees Created 


Metrics to monitor process 
execution that also include an 
export to Microsoft Excel 
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Tools (3 of 3) 
R6o0 - Raytheon Six Sigma 


=» R6o is company 


culture — spans 
business units 


EVERYONE is 
trained and 
qualified as a 
specialist 


Raytheon Six 
Sigma is part of 
the DNA 


There is a robust 
set of tools and 
trained experts 
to facilitate their 
use 


Raytheon 


Network Centric Systems 


Back xy > ~ Q ft | QSearch GyFavorites —PMedia <4) SS~ [A y 5} 
Addiess =] Go | Links 


Raytheon Six Sigma Home Roc Supply Chain Business Unit Focus 


Master Experts: 


Raytheon Six Sigma’ 


Deve bpme tt Process 


Coachlig Process 
Eipe rt tide the 
Usberse 
CarreatCarricuam 
Prereqaiste Checkiist- New 
7 kee 
Certification Process 
CortectSteet- New 
Tralbhg Arck te 


$S0 PasswordProblems? 
ISSSP Password 


2 * 
R60"Online NeWS j nore... 
13-Oct-06 Congratulations September 2006 Raytheon Six Sigma Experts! 


Specialist: 


Project Pia Tempe 
ebastamarnee 03-Sep-06 Congratulations August 2006 Raytheon Six Sigma Experts! 
st ‘ 29.Auq-05 2006 Raytheon Six Sigma Celebration Coming in 
SpeckistTrahhg Seem 
Specwikt Profect: ab 
STS Os. POCs 


R6o Quick Reference 


> heon Six Sigma™ Official 


Usage Guidelines 
October! > R6o Scorecard Metrics 


> Log Into POWERSTEERING 


21-Aug-06 Congratulations July 2006 Raytheon Six Sigma Experts! 
Project Librar: Help. 


STS Hep Certer- Mew New Guidelines for Raytheon Six Siqma™ Use Issued > Knowledge Management 
STS Fans ® 
(US Domestic access ONLY) R6c Now Registered Trademark of Raytheon Company > IPDS 


Popular R6o Tools Toolt Matric. 
@easy QDmoderate @ difficutt 


Getting Started 
> What's In it For Me? 


R6O Learning Events 


> Raytheon Learning Accelerator 


Events & Dates 


Bpertce tiation Hoands- > How To Start A Project? (PLA) @ Project Plan Template 
- matey F >FAQ's >R6a Practitioner Tracks @ 20 Keys 

wpe rtc basses 

Soledele 2005 - New > Project Plan Template >R6a for Design @SPC XL (mor srcinto) 
Sohedtle ZT New > Six Sigma Project Presentation © > Enabling Business Growth (EBG) @5 Why's 

Cakidar Amr - New Template e DOE PRO 


@IPO Diagram 

@ Logical Process Map 
@VTP-Visionary Team Planning 
Popular Downloads 


R&c Corporate Calendar > Points of Contact > Integrated Supply Chain 


> Critical Chain Project Management > Program Performance 
CCPM) 
> Raytheon Six Sigma Orientation 
Hote: NSIE Qe netEpbreh required D the => Project Presentation Template 
couse. 


v 
EY Pinca then nese OAL. — Nractitionor Track Drachuro fh | 


i] ay [ey rey (GE Local intranet 


L1/10/ZUU0 | Paget 


> Support & Enable Process 
>R6a Accelerator Workshop 
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Communication Briefings 


= All have a common format Conon oes wher 
= They are targeted to be self led 

; Organization Improvement 
= They are posted to the web site for Process 


easy down load on demand 


Network Centric Systems 
Common Process Architecture 


Network Centric Syst 
Agenda Common Process Architecture Organization Feedback 
- wee Network Centric System: 
~ Why? Common ae eae 
- Who? 
2 "Fadleqieed ly How to Submit an Organization 
siBehavicns used & Improvement Proposal 
- Process 
~ Available Help - \) Network Centric Systems 
os Summary 3 Ps ] Common Process Architecture 
- Expectations 


How to Evaluate an Organization 
Improvement Proposal 


oa 


“QS Improve It! 


Communications Briefing - Organization Improvement Process U0141HOT Rev — 11 Sep 2006 


It is impossible to over communicate 
11/16/2006 | Page 14 
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People 


NCS Process Organization Structure 


NCS Engineering, Technology, & Quality Leadership Team (ETQ LT) 


ae ; Lynn Dugle (VP) 
NCS Engineering Councils Process: Rich Karm ++ Ft. Wayne Eng: Ken Steiner 
PE: Dan Robinson (lead) + Business Ops: Suzanne Hays Florida Eng: Todd Trapp 
SE: Leif De Wolf (lead) + Eng Automation: Dan Starcevich + Fullerton Eng: Andy Jansen 
SW: Lynn LeBlanc (lead) t R60: Suzanne Hays Fullerton TRS Eng: Bill Pye 
HW: Pete Fleming (lead) + Quality: Mike Gault ¢ NE Eng: Scott Whatmough 


QE: 
Site Process Leads 


Mike Gault (lead) + Technology: Kevin Riley Texas Eng: 


Organizational Improvement Coordinators (OIC) 


Lead: Michele Wall 


Joe Castellino (acting) 


CPA Lead: Tony Delaporta + 
NE: Fed Cox t+ 

Texas: Maureen Wampner + 
St. Pete: Tim Wellman + 
Fullerton: Dennis Laiola + 

Ft. Wayne: Karl Davis + 


NCS Measurement Council 
Lead: Ann Laviolette (acting) 
Technical Director: Sally Cheung Deputy Technical Direct: 


ETO Learning Team NCS Process Organization Structure 


Lead: Kathy Nogues 


CPA Configuration Control Board (CCB) 


Chair: Janet Bratton . . . 5 
— En NCS Engineering, Technology, & Quality Leadership Team (ETQ LT 


CPA Tools Configuration Control Board (C Gounals Lynn Dugle (VP) 


Chair: Dawn Spiri 


oi = a a 
ee ; see ale Organizational Improvement Coordinators (OIC) 
f Engineering Process Steering Lead — NCS OIC: Michele Wall (NE) 
Team (EPST) Member CMMI® Appraisal Team Ft. Wayne; Rick Warmels Fullerton: Laurie Haack NE: Lynn Bailey 
Tt EPST Lead Lead Appraiser: John Ryskowski (JFR Consulting, Lead St. Pete: Deb Palmer Texas: Becky Smith EA: Dawn Spiri (EA) 


Site 
Process 


Leads NCS Measurement Council 
Lead: Ann Laviolette (NE, acting) 


4 Technical Di > Sally Ch (Full ) D Technical Di : Steve Allo (Texas) 
The OIC Role is the only FuWayne: DeusErk =~ Eullnton! SiexéHellSSSONE! EA Winchan 
- eae St. Pete: Nancy Hunter Texas: Chris Angermeier 
improvement process addition. 
A ETQ Learning Team 
: The OIC lead IS a R60 Expert. Ft. Wayne: Brian Davis Renae ee ee NE: Philip Pirello 
i St. Pete: Cheryl Elko Texas: Debbie Hoff 
¢ The OIC Team includes a i 
cl c CPA Configuration Control Board (CCB) 
representative from each major Chai: fane Baton 
CCB Configuration Mgr: Becky Smith CCB Quality Engineer: Bruce Riggs 


site. All OIC members review ee See oe Sw: Any Donahue 
SEY 
proposals. 
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Org Improvement Annual Planning 


ETQ Delta Gap 1st Pass 2nd Pass Package | Gatea’s | 
Kickoff Analysis Roadmaps Roadmaps Rev/Rank 


Select 


Analyze Step 1 Analyze Step 2 Analyze Step 3 Sart 
¢ Draw Zot line 


* What is it * Approach * How does the se 
¢ Cost project measure up * Defend decision 


¢ RO! against the 
evaluation criteria 
¢ What is the 
Stakeholder 
viewpoint of 
importance? 


¢ Case for Action 
¢ Benefits 
° Synergy 


We have a strong Annual Planning Process 
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Results to Date 


Notional project 


sisal Review Eval 
TYPE |TITLE LEAD Board a Area Score 
Appr development technology Pochahan 
° There are over 40 rorase a ecu and over { 
3 [ow | “cor | i 
Appr Hangon * Data in the form of business, quality and process 
+ | npr | mprfhosten conser | performance metrics trigger a variety of projects i 
i 
i 


| 5 | Appr|  /  \proveTest_ | ¢ Projects that address handshaking between 


[ae || alll functional organizations 
Appr isfaction . 


¢ Projects that target defect containment in key 
ae ia development stages 
¢ Projects that address insertion of technology and 
methods that require challenging paradigm shifts 
¢ Projects that are not gap closure - but opportunity 
grabbing 


Projects are Executing in all disciplines and focus areas 


eee ou 


ige 17 
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Customer Success Is Our Mission 


Overview 
Solution 


v2 Journey 


Lessons Learned 


m 
Network Centric Systems 


Where did we want to go? 


=» Move from separate improvement processes across sites, various 
disciplines to integrated improvement 

= To get improvements in engineering performance that are reflected 
in the business bottom line 


NCS NE 
EPG OID 


EPG OID 


NCS CA 
EPG OID 
NCS TX 
NCS FL 


EPG OID 


EPG OID 


Scope is local. Goals are local. Scope is NCS. Goals are NCS. 
Process Is local. Metrics are local. Process is NCS. Metrics are NCS. 
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Challenges/Barriers Raytheon 
k 
Hold on Sherlock...it’s not so easy Network Centric Systems 


~4000 engineers, 5 sites, multiple time zones ... 
and 5 disciplines at each site!!! 


¢ What - no S 
face to face — 


will never » I need you for this and — they can’t know my 
work needs 


| am here now...that 


~@) « That's not ' ° 
BMD | winat we do IP lead isn't ond 
¢ Improvements aren't ; ; 
and we’re urgent ¢ Single point of 
level 5 decision at so high a 
level that details of 
¢ SW,HW,SE,PE, program needs improvements are 
are different...functions versus difficult to assess 


programs 


¢ It can’t cost more 
¢ The process, infrastructure and tools may 
not scale well 


The challenge of the org 
The challenge of the tools 
The challenge of the process 
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Raytheon 
The Journey Network Centric Systems 


Build a team and identify stakeholders 

Align the team on a common vision an 
Ensure team commitment mia 
Take stock of regional assets BS 


Take stock of company assets Bie. 
Identify common components and Ao 
use a DAR type process to look at es: 
alternative solutions — address — 4} — av 
difficult questions first | ci Bl he ee eae 
Develop and Deliver a Concept Brief = |_ NV | | mens 

lterate until concept is approved eS. Mer Nt on. teen, 
Develop artifacts 

Peer review artifacts with team 

Submit artifacts for stakeholder review 


Release 
DEPLOY 


Ooovovo oO 


Ooocvovcoo wo 
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=. Raytheon 
Align the Team on a COMMON VISION network centric systems 


ID: NCS EPG 

Propose: NCS EPG (OID WG) 

Prioritize: NCS EPG 

Rank: NCS PST annually 

Approve$: NCS PST 

Monitor: NCS PST gtrly 
Assigned org monthly 


Scope is NCS. Goals are NCS. Gaps are NCS ETQ. Process is NCS. Metrics are NCS. 
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Take Stock of Regional Assets 


a Solution 
Components 
-R6S Toolkit 
. -Blue Book -R6S Template 
— Work Instruction -SEPG-020 Eval Template Web Stas ~ | cepeces. ie ee 
SEPG-002 Pilot Templat pa ere ee 
; i elas -web submit -Project Lead 
— Te mM plates -Roles/Resp -Deploy Template -WSTR DB 
— Tools 
-RFO 5-2 
= In| -Bulletin 5,13-15 
Training ons 


-REO'5-2-5 


-Appropriate 
-Form 21832 : leader 
PO M-RFOS-2-7-201 | onc a a0 Website —|- Petaie 


— Review Boards IRFO 5-2-6 
— Approval Boards 

-RFO 5-2-8 -Excel Log 
— Documentation Poe 


-RFO 5-2-4-104 
-RFO 5-2-4-105 
-RFO 5-2-4-204 


rREO 5-2-7 -R6S Toolkit 


Baseline 


-SW Dir-000 -R6S Toolkit 
-800P105 -Plan and Report -web status 

-800P020 Template -web submit 
-800P019 -P| DB 


-SEPG 
-SWEC LT 
-Tech Team 
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Analyze Regional Assets for 
Commonality and Potential Reuse 


NCS NE 


NCS FW 


ID: SEPG CCB 
Propose: PTIC 

(Process Technology 
Insertion Committee) 
Rank: PTIC constant 
Approve$: SWEC LT 
Monitor: PTIC + LT 


| 


ei ) \ 
ID: EPG CCB — 
Propose: EPG OID ID: Sere te ero 
Rank: Baseline Assessment ie aed T ne wi 
Approve$: LT + Technology Rank: LT + Technology 


Monitor: EPG OID, LT ApproveS: LT 
Monitor: OIB 


Sites all had mechanisms to submit ideas, approval boards, and coordination POC. 


Differed in tooling and selection cycles. 


mn 
Network Centric Systems 


Take Stock of Company Assets 


» FU, TX and NE all leverage Raytheon Six Sigma which include basic 
steps below 

» Leveraging happens within site organization and cross sites 
through papers or symposiums - but after the Evaluate, Pilot and 
Deploy steps 

» R6o provided a common look at feel to separate regional processes 


R6S Step 
Visualize 
Commit 
Prioritize 

| Characterize 
Improve 


Achieve 


Everybody utilizes R6o - TREMENDOUS lever and common communication 


vehicle ge 25 
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Which way to go? 
a Lift, Modify or New Improvement Approach? 
— Should one of the regional approach’s be adopted, is a 
blended approach needed, or a new concept entirely? 
» “Distributed” versus “centralized” organization 
improvement? 


— Distributed - Regional level improvement activities with 
added communication mechanism between regions g 


— Centralized - NCS-level organization improvement ~% 
activities 


— Hybrid of both? 
— Discipline based? (HW, SW, SE, etc.) 
— Focus area or theme based? 
a Who should be part of the organization 
improvement decision making/leadership? 
— EPG 
— Regional or NCS level Improvement Boards 
— Engineering Leadership 


11/16/2006 | Page 26 
Copyright © 2006 Raytheon Company. All rights reserved. 


m 
Network Centric Systems 


Formulate alternatives and analyze them 


=» How to decide? 
—Used decision analysis techniques and 
considered: 
- ROI 
e Natural organization structures 
e Alignment with organization goals 
e Cost effective 
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Option 1 - Raytheon 
Regional Improvement Boards Network Centric Systems 


Joe Engineer 
submits idea 


Research 
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Option 2 - Raytheon 
Discipline Improvement Boards iittala 


Joe Engineer 
submits idea 
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Option 3 - Raytheon 
Theme Improvement Boards MELEE SENS a nSCH 


Joe Engineer 
submits idea 
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Raytheon 


Network Centric Systems 


NCS Process 
Steering Team 
(PST) 
¢ Process Director 
a Oxo) lalei| M@varclies) 
oe ay Wa D) | gsxeire)e 
Process Leads 


LZ’. 
| 
| __&__ | 


ESCM teacersnip] CCB | Metricg] OID [Jtools | PAL | SMEs | ... [complance SN RCC Cai 
foo (_ 


ESCM Ueacersnip| CCB | Metriod | OID [JToos | PAL | SMEs |... [complance EN Rec Cea 


Pe a ee 
SRS eecersne | CCE [ Wewied] O10 [foots [PACT Ses | [conptence SISSIES 


ann) nee 
ROME (coco | CCS] Metros) OID Tools | PAL | SMES]... [conpiance EIRICONEEN 
es es ee, Ys 
ESCM ceacerstip] CCB ] Metres f OID Tools | PAL | SMES |. [complance SN RCCnCn am 
\ J 


Identify the Stakeholders 


ETO LT 
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Raytheon 
Pp arad Ig m Sh ift Network Centric Systems 


=» Move improvement activities out of the jurisdiction of the 

regional EPGs and into the jurisdiction of the organization 
leadership. 

— Better alignment with org level objectives 

— Stronger sponsorship 

— Less redundancy 

— Better communication across regions 

— Better ROI 
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Customer Success Is Our Mission 


Overview 
Solution 
Journey 

»—> Lessons Learned 


Lessons Learned 


m 
Network Centric Systems 


Use the CMMI as a common language to facilitate understanding one 
another’s solutions 


Use OID concepts to create and deploy org improvement 
The metrics are key — at the beginning, in the middle and at the end. 
Build on what you have 
— Wherever annual planning goes on — and everybody has annual planning, - 
that is collecting and selecting improvements 
— Improvements need sponsorship so make it the responsibility of leadership to 
review and select projects. It's where the money and resources are. 
You need a collection tool and it might as well be a communication tool 
too. It should be public. 


Cannot talk about OID as just the project collection, selection and 
execution piece, it is the loop between 4 and 5 that is key and all those 
processes work together. 


Get some things straight right away: Will every project that improves 
anything be part of this — what is the scope of an improvement? What is 
the relationship to CAR? What is the relationship to Technology? 

There is significant impact to appraisal preparation 
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Words from our leader.... 


“Our approach to process commonality is already reducing the 
process maintenance and appraisal costs, and we have 
invested those savings in new engineering capabilities and 
growth opportunities. Consistent processes and better 
tooling/ automation enable us to respond faster with lower 
costs, increasing our competitive edge.” 


Lynn Dugle, Vice President of Engineering, Technology and Quality, NCS 
Fullerton 
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Raytheon 


Network Centric Systems 
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Acronyms 


Acronym Definition 

CPA Common Process Architecture 

DAR Decision Analysis and Resolution 

EA Engineering Automation 

EPG Engineering Process Group 

EPST Engineering Process Steering Team 
ETQ Engineering Technology and Quality 
FW Fort Wayne, Indiana 

HW Hardware 

IPPDB Improvement Proposal/Project Database 
LT Leadership Team 

MSI Mission Systems Integration 

NCS Network Centric Systems 

NE North East 

OIB Organization Improvement Board 

OIC Organization Improvement Coordinator 
OID Organizational Innovation and Deployment 
PST Process Steering Team 

R6s Raytheon Six Sigma 

ROI Return on Investment 

SE Systems Engineering 

STP St. Petersburg, Florida 

SW Software 

WG Working Group 

wi Work Instruction 


Raytheon 


Network Centric Systems 
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NORTHROP GRUMMAN 


DEFINING THE FUTURE 


ys 


-‘Making OID 
: e Effective 
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= What ts “Effective” 
= Where We Struggled 
= What Had to Change 


= Measures of Change 


YORTHROP GRUMMAN 


Copyright 2006 Northrop Grumman Corporation 


Northrop Grumman Mission Systems 


Mission Systems Sector has Intercontinental 


23,000 employees in 9 a Ballistic Missile 
Program 


divisions 
= Received first CMMI Level 5 
| SCAMPIA rating in April 2003 


= By the end of 2006: 
Joint National 


25 externally-led CMMI Integration Center } 
Level 5 SCAMPI As 


99 projects through CMMI 
Level 5 SCAMPI As 


Hundreds of CMMI Level 5 
SCAMPI Bs and Cs 


Satellite Command & Control 


WW Patleum A) ean ee ea, 
NWMORT FIFO 


iJAd Ala 
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ee 


Copyright 2006 Northrop Grumman Corporation 


How Can We Better I nstitutionalize OI D 
Behavior? 


SCAMPIAs |____2003, 


OID Practice Ss HE =f als a 
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How Do We Know OID Isn’t I nstitutionalized? 
It Doesn’t Feel Right 


1), SCAMPI A 
means It’s 
time to 
worry, and it 
shouldn’t 


3). Not many 


2) Not many 
improvements 
are submitted 

each year, which 
Is only “ok” 


>= 


projects are 
finished each 
year, which is 
again, only “ok” 


NORTHROP GRUMMAN 


Copyright 2006 Northrop Grumman Corporation 


What Did We Do? 


r = Overhauled the OID process 


= In February 2006, completed an OID project called: 
“OID Revamp, i.e., OID of OID” 


NORTHROP GRUMMAN 
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Ww OI D Refresher 


3 
— tinh a 
ff =} CF | Select yp SG 2 Deploy 
a, improvements improvements 
SP 1.1 Collect and Analyze SP 2.1 Plan the — 
. Improvement Proposals Deployment — 


SP 1.2 Identify and SP 2.2 Manage the 
PNariMy {=m lalatencclace) ary Deployment 


SP 1.3 Pilot Improvements SP 2.3 Measure 
Improvement Effects 
SP 1.4 Select 


Improvements for > 
Deployment 


6 ef 
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Level 3 OPF versus Level 5 OID 


OPF OID 


= Goals are qualitative =" Goals are quantitative (e.g., reduce 
(e.g., get better) variation by X% and/or mean by Y%) 


= Effects of improvements = Improvements cause a shift in 
are not estimated or process capability, i.e., performance 
measured and/or quality 


= Potential improvements are analyzed 
to estimate costs and benefits 


= Improvements are piloted to ensure 
success 


= Improvements are measured in terms 
of variation and/or mean 


NORTHROP GRUMMAN 
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Number of OI D Projects is Only “Ok” 


M TW T F 5 5 M TW T F 5 5 M TW T F 5 5 

l¢s 4 5 l2@a24 5 1 ¢sa 4 5 
6 ¥ @ 91011 le 6 ¥ &@ 91011 le 6 ¥ @ 91011 le 
13.1415 1617 1619 13.1415 1617 1619 13.1415 1617 1619 
20 21 #24 23 24 25 26 a0 21 #4 235 24 25 26 20 21 #4 235 24 25 26 
2? 20 29 30 31 2? 20 29 30 31 2? 20 29 30 31 


e.g., Project Plan e.g., Tests for 


e.g., Electronic 


Templates Training Courses Evidence Tool 
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Actual Goal of the “OI D of OI D” Project 


The previous OID process was not 
effective and resulted in very few 
completed improvements per year. 
To be effective, approximately 5 to 
10 improvements should be 
completed per year. The measure 
of improvement will be the number 
of suggestions provided per year 
and the number of improvements 
completed per year. 


NORTHROP GRUMMAN 
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Change 1: 
Changed the Organizational Structure 


Level 4-5 
Working 
Group 


NORTHROP GRUMMAN 
10 
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Change 2: 
Assigned Clear Responsibility 


Awards 


= . Mone 
a’ Matt 


Raises —_— 
"eH Performance 
Reviews 


Hmm,... which 
should | work on? - 


rN fe % 
QYSs 


% 
—_ L 
= WG y 
Aye 


Thank You 


NORTHROP GRUMMAN 
at 
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Change 3: 
Became More Proactive 


Reactive Proactive 


(Wait for Suggestions) (Search for Suggestions) 


Divisions 
& Projects 
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Change 4: 
Eliminated the Bureaucracy 


ay 
-X Simplified Six Sigma 
templates for OID, 
. eliminated tollgate 
reviews, etc. 
| Simplified the 
uphill battle. 


NORTHROP GRUMMAN 


13 Copyright 2006 Northrop Grumman Corporation 


Change 5: 
Avoided the Term “Innovative” 


One New 


ficeina Process Step Salih 
Error A Series 
Spelling of New 
Error Templates 


Bade 


Learning 
Management 
System 


= There are a wide range of improvements 


= Projects vary from very simple changes to 
complex systems 


NORTHROP GRUMMAN 
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Change 6: 
Resurrected the Watch List (1 of 2) 


Watch List was not E Let the Watch List 
used frequently loose for visibility 


Resurrected 
the Watch List 


NORTHROP GRUMMAN 
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—— 


| Change 6: 
Ww Resurrected the Watch List (2 of 2) 


The Watch List is 
statused monthly 
at the CCB and to 
higher-level 
management. 


Watch List Fields 


16 


Change 7: 
Made Successes More Visible 


wv 
2 SHR — Z 
Hey, | should 
submit the 
tool | created. 
e 


NORTHROP GRUMMAN 
17 
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Look at Measures NOW 


cea at the measures — 
n= ut = 

2004 In=3 for 2006 2004 Out = 2 
2005 In=6 2005 Out = 3 


e.g., Software Product Lines 


NORTHROP GRUMMAN 
18 


Copyright 2006 Northrop Grumman Corporation 


What Still Needs to be Fixed 


NORTHROP GRUMMAN 
19 
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hanged the 
Organizational 
Structure 


Made 
Successes 
More Visible 


Assigned 
Clear 
Responsibility 


Resurrected Became 
the 


Watch List 


More 
Proactive 


Avoided Eliminated 
the Term the 
“Innovative” Bureaucracy 


NORTHROP GRUMMAN 
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TALI ISRAEL AVRCRAFT INDUSTRIES LTD 


as. 
INITIAL LESSONS LEARNEC 
FROM CONDUCTING SCAMPI-B 


APPRAISALS 


ITZHAK LAVI 


SARIT ASSARAF 
ISRAEL AIRCRAFT INDUSTRIES LTD. 


CMMI Conference —Denver Nov. 2006 


Initial Lessons learned from conducting SCAMPI-B appraisals 1 I, LAVI S. ASSARAF — IAI Nov.2006 


AIRCRAFT INDUSTRIES LTD 


= * 
@ Background 


@ IAI Process Improvement Path 


@ CMMI Implementation in IAI 
@ SCAMPI B&C Appraisals 
@ Performing SCAMPI-B Appraisals in IAI 


@ Lessons Learned and Recommendations 
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israel Aircraft Ii 


@ Largest industrial company in Israel 
@ 15000 employees, 2.3 $B annual sales 


@ Involved in Development, Production, Maintenance 
and Service of Aerospace Systems 


@ IAI divisions are ISO9000 and AS91100 certified 


@ Started SW-CMM based Software development 
process improvement efforts in 1992 


@ Started CMMI based process improvement efforts in 
2002 
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Y/. RAEL AIRCRAFT INDUSTRIES LTD 


{Israel Aircraft Industries Groups 


ELTA Space &, Military Conmmnenciialll 
Missiles; Alircranfits; Ali cranfits; 
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3 Ability Improvement (CAT) 


: Production Process Improvement 

J AVeejtiisiites Pree MprOvEMeNnL 
| ~<a ¢ Increase 
Pr mm Felavslels)aalcl alae ca melerstotom lanl e)aet.s) patel als > = Quality 


° ” Proje oject Piennine g an: d( Ca oni ntrol 


¢- Risk Management 
e Increase 


Sales 


NCESVA =) cole l Ofer me |alumeyelUlera(o) am (Nmap) 
IAI Uniform Project Process Framework e Increase 
Profit 


Engineering Development 
aie ogies 


ae 


0) eae L AYRCRAFT INDUSTRIES LTD esseF A 
Israel Aircraft Industries Process Improvement Path 1 


Starting activities for Starting CAI 
project management activities and NPI 
process improvement development 
1993 1999 


Starting SPIP — CMM Identifying System Stated SSI=F cM 


based Software Process Engineering process 
Improvement Program needs and starting 
1992 SYSPIP 1997 


20 00 
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Israel Aircraft Industries pre 


@ Process improvement efforts are coordinated at the corporate 
level and sponsored by the company VP of Operations 
@ Many infrastructure assets are developed at the corporate level 


and are adapted by the groups and divisions. Other assets are 
developed within the groups and divisions 


@ Process improvement implementation is conducted at the group 
and division level and sponsored by the group or division 
management 
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SlAl ISRAEL AHRCRAFT INBISTAIES LTD a 
Israel Aircraft Industries Groups and Divisions 


AI 


ELTA Space & Miiillittanry, Conmmenciiall 
Missiles; Avie ranfits; Aline ranfits; 


ELTA 
AEW Engineering 
IMINT and Radar Aircrafts _ 


ELTA 
SIGINT, EW and 
Communication TAMAM 


ELTA 


Technologies MLM 


Hl CMMI Based Process Improvement Activities 
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Appraisals in IAI 


@ Since the beginning of the CMMI implementation 
program in 2002, there have been many formal and 
informal process reviews and appraisals in IAI groups 
and divisions, with the CMMI as the reference model 
with the following objectives: 


% Identify strengths and weaknesses in the defined 
processes 


% Identify strengths and weaknesses in the way processes 
are actually performed 


@ Provide the basis for process improvement activities 
@ Determine CMMI maturity level 
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The need for a formal non SCAT 


@ While SCAMPI-A appraisals provide a reliable way 
the CMMI maturity or capability levels, they are not ; 
identifying strengths and weaknesses during the process 
improvement effort 


@ We needed a method that will be: 
@ Well defined and documented 
- Can be applied consistently 
@ Flexible and tailorable 
- Can be applied in many different situations 
Less effort than SCAMPI-A 
- Can be applied more frequently 
% Will not provide rating 
- Will not create the stress usuallyoassociated with SCAMP 
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AMPI-B Appraisal | 


@ Defined at the SEI in 2005 as ARC-B compliant - 
appraisals method 


@ Purpose: To appraise a sample of the processes 


defined and_ implemented in the organization, and 
identify strengths and weaknesses, WITHOUT 


assigning maturity or capability level ratings 


@ Provides a higher level of flexibility than SCAMPI-A 
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@ Define appraisal scope: 


@ The organization parts to be appraised 


@ The CMMI model scope to be considered 
@ Prepare formal Input and Plan documents 
@ Collect information from both direct artifacts and affirmations. 

® Indirect artifacts are usually not used in data sufficiency rules of SCAMPI-B 
@ Use data sufficiency rules and Draft Findings to verify and validate results 


@ Characterize practice implementation level as Red, Ye!!ow or Green to provide 
easy understanding of the organization process status 


@ Appraisal team must consist of at least 2 members, in addition to the team leader 
@ Non Attribution 
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AMPI-C Appraisal | 


@ Defined at the SEI in 2005 as ARC-C compliant : 
appraisals method 

@ Purpose: To appraise a sample of the processes 
defined or planned in the organization, and identify 
strengths and weaknesses, WITHOUT assigning 
maturity or capability level ratings 


@ Provides a higher level of flexibility than SCAMPI-B 
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SCAMPI-C A 


@ Define appraisal scope: 
@ The organization parts to be appraised 
@ The CMMI model scope to be considered | i 


@ Prepare formal Input and Plan documents 


@ Collect information from direct artifacts, indirect artifacts and/or 
affirmations 


@ Data sufficiency rules are less firm than in SCAMPI-B 
@ Non Attribution 
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Al Goals in 


@ Use a well defined, repeatable and fle 
process appraisal 


% Obtain information about current processes 
¢% Identify good practices that should be repeated 
% Identify weaknesses as early as possible 


% Obtain information about the level of readiness for a SCAMPI-A 
appraisal 


@ Avoid the stress associated with SCAMPI-A 
@ No rating 
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SCAMPI-B Appraisals Pert 


| 7) ; 
@ Two formal SCAMPI-B appraisals were conductec in IAI: 
@ 1st SCAMPI-B appraisal performed in IAl LAHAV division in 
September 2005 
@ 2nd SCAMPI-B appraisal performed in IAl ELTA group in 
December 2005 
@ In addition, three informal SCAMPI-B like appraisals 


were conducted in the same two organizations in 2006 
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Initial Lessons Learne 


@ Scope must be defined very carefully, especially when the time is _ 
very limited (5 to 7 days at the most) 
@ Usually: 2-3 projects, 4-5 process areas 
@ Preparations for SCAMPI-B are similar to those for SCAMPI-A, e.g.: 
@ PIIDs preparation 
@ Appraisal team selection and training 
@ Plan, Logistics, Schedule 
@ Interview Scripts 
@ Draft Findings sessions may be less important than in SCAMPI-A 
@ For frequent PIIDs review SCAMPI-C may be more appropriate 


@ SCAMPI-B appraisals helped in focusing management attention 
and raising awareness of important process issues 
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Initial Lessons Learned -2 


es _ — 


Lom 
| ‘ 


@ Best results are obtained when the team consists of people from 
both the appraised organization and from outside 
@ SCAMPI-B may not be appropriate for initial process appraisal, 


because of the need to consider direct artifacts. SCAMPI-C may 
be better for this purpose 


@ SCAMPI-B may not provide enough information about readiness 
for SCAMPI-A because Indirect Artifacts might not be examined 
ina SCAMPI-B 


@ In some cases, SCAMPI-B method does not provide enough 
flexibility 
® Draft Findings Presentation contents 
@ Combinations of project and CMMI scope 
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Combinations of 


PP PMC REQM CM 
! rN / N y \ 
/ h 

Project 1 DA F DA \ / DA 1 flare ll 
/ \ | \ 
; | \ l 1] | 
Project 2 DA {DAT 1 }oalup tac | 4 
[ il l 
\ | | in| | 
Project 3 Ae ee], 1 |DAlpular | 4 
\ | \ I \ | 

Project 4 \ ! \ I \ ! 
oa \ } \ Ee AF J 

y; | v4 

DA = Direct Ars#4cts = AF=Affirmations 


Sufficient Data 


Insufficient Data = = = 


Data Sufficiency rules require: 
eAt least one direct artifact or one affirmation for each instantiation in scope 


eBoth a direct artifact and an affirmation when considering all instantiations in scope 
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Combinations of Proj 
ot considered_in S¢ P|-E 


ee -_ 


% EP PMC REQM CM 
Project 1 
Project 2 
Project 3 
Project 4 
|__| In scope [| _] Out of scope 


Some process areas may intentionally not be covered in some projects to 
make better use of appraisal preparation and on-site time 
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Combinations of Proj 
ot considered in S(¢ D|-B 


: = -_ 


yPP 
Project 1 
Project 2 
Project 3 
Project 4 

|__| In scope 


PMC 


REQM 


| _]Out of scope 


CM 


Some practices may intentionally not be covered in some projects to 
make better use of appraisal preparation and on-site time 


Initial Lessons learned from conducting SCAMPI-B appraisals 


20 


I. LAVI S. ASSARAF — IAI Nov.2006 


pe 


[IAT ...- - 
WEEE |SRAEL AIRCRAFT INDUSTRIES LTD 


Recommendations regardine 


Use SCAMPI-B appraisals systematically as one of the tools ¢ n the process 
improvement path | 
@ Perform once a year to check process status between SCAMPI-A appraisals 


@ Limit the scope to one process area category (e.g.: Project Management PAs, 
Engineering PAs) 
@ Plan SCAMPI-B appraisals in conjunction with other appraisals (Such as 
SCAMPI-C) so that maximum information is obtained with a minimal effort 
Use SCAMPI-C appraisals for more frequent process and PIIDs preparation 
status review 
Make sure that the team consists of people both from the appraised 
organization and from outside 
Make sure that expectations from SCAMPI B & C are aligned with the methods’ 
capabilities 
@ SCAMPIB or C are not a cheap replacement for SCAMPI-A 
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TAT iSRAEL WIACRAFT INDUSTRIES LTD 


Periodic Formal 
Appraisals to 
determine Maturity or 
Capability Levels 


identify Strengths 
and Weaknesses in 
Processes 


Periodic Appraisals to 7 


Q1 


Sample plan involving : 
—— 


Q3 | O04 


Interim evaluation of 
progress in process 
improvement and 
readiness for 
appraisals 


A SCAMPI-A 
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A SCAMPI-C 
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Recommendations regardin 


Modify the SCAMPI-B method to eni 
flexibility 
- Scope definition 
- Draft Findings presentation 
Find a way to consider indirect artifacts in SCAMPI-B 
data sufficiency rules, in addition to direct artifacts 
and affirmations 
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Projects for an Appraisal 
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¢ At the 2005 NDIA CMMI and SEPG 2006, we presented 
“Building a Credible SCAMPI Appraisal Representative 
Sample”. 

¢ This presentation demonstrated how to apply Design of 
Experiment (DOE) techniques in choosing a “good” 
SCAMPI appraisal representative sample. 


Today’s presentation builds on our previous work to: 

- Provide advice on how to select projects for a SCAMPI appraisal 
while following the rules of SCAMPI Method Description 
Document version 1.2. 

- Demonstrate applying the more mathematical of the project 
selection techniques using Microsoft Excel. 
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Picking the Right Projects for an Appraisal? 
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¢ Choosing the right projects is the foundation of an appraisal that: 
- Iscredible to anyone who was not part of the appraisal team or even 


part of the organization being appraised and 

- Maximizes the organization’s understanding of its processes’ strengths 
¢ These two benefits of choosing the right projects are mutually 

- Choosing the right projects provides the most information gained per 


and weaknesses. 
appraisal resource invested. 
- Other sets of projects could be examined, but would be inferior with 


supportive because: 
respect to insights gained on process institutionalization. 
a less-than-right set of projects invites suspicion that the 
results may reflect projects that are atypical of the processes 
eu projects maximizes the return on 


apprais 
¢ Atypical because only “good” projects were included or 


- cross 
a 
used in the organization. 
¢ Atypical because “bad” project were excluded. 
¢ Bottom Line: picking the ri 
investment in the appraisal 
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¢ Section 1.1.3, “Determine Appraisal Scope’ : 


- The organizational scope of the appraisal must 


include sample projects and support groups that are 


representative of the implemented processes and 
functional areas being investigated within the 


organizational unit and that operate within a 
coherent process context. The rationale for selecting 
these sample projects and support groups as 
representative of the organizational unit must be 
documented. 
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| Picking the Right Projects? (2) 


¢ Section 1.1.3, “Determine Appraisal Scope”: 


- Organizational unit size (i.e., number of people and number of 


projects) and sizes of projects and support groups (i.e., number 
of people) in the organizational scope must be documented as 


well as the percentage ratio of these two measures: 
¢ Population %: the number of people in the organizational scope 
divided by the number of people in the organizational unit (x100) 


° Project %: the number of projects in the organizational scope 
divided by the number of projects in the organizational unit (x100) 


- Critical factors that influence implementation of practices in 
projects and functions within the organizational unit must also 
be understood and identified. 
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¢ Section 1.1.3, “Determine Appraisal Scope’ : 
- Sample projects and support groups selected to form 


the organizational scope (i.e., the combination of 
focus and non-focus projects and support functions) 


must represent all critical factors identified for the 
organizational unit to which the results will be 


attributed. The coverage of the organizational 
critical factors provided by these sample projects and 
support groups in the organizational scope in relation 
to the organizational unit must be documented, in 


quantitative terms, in the appraisal input and ADS. 
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¢ Section 1.1.3, “Determine Appraisal Scope”: 


- Each sample project or support group in the planned 
organizational scope of the appraisal must be one of 
the three types listed below: 

¢ Focus projects must provide objective evidence for every PA 


within the model scope of the appraisal which addresses 
model practices applicable to those projects. 


¢ Non-focus projects must provide objective evidence for one 
or more PAs within the model scope of the appraisal which 
address practices performed on projects. 


- In appraisals where the reference model scope 
includes any project-related PA, the organizational 
scope must include at least one focus project. 
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¢ Section 1.1.3, “Determine Appraisal Scope”: 
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- Information about the organizational unit should be 
documented in a way that allows future appraisal spansors to 


replicate (to the extent possible) the organizational unit 
appraised. This information should be in the appa plan, and 


used (in summary form if needed) in briefing the appraisal team 
and appraisal participants. Other examples of critical factors 


include: 
application domains (or lines of business) 
geographical breadth 


disciplines (e.g., systems engineering, software engineering, or 
hardware engineering) 


effort types (e.g., development, maintenance, or services) 
project types (e.g., legacy or new development) 
customer types (e.g., commercial or government agency) 


lifecycle models in use within the organization (e.g., spiral, 
evolutionary, waterfall, or incremental) 
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¢ The MDD does not advise us on how to pick projects, 
but it does provide conditions that our selected projects 
must fulfill to be acceptable: 


1. You must document the percentage of the organization's people 
and projects covered by the appraised projects and functions. 

2. The appraisal must use representative sample projects. 
Representative means that all critical (process implementation- 
influencing) factors are covered in the sample projects. 

4. The relationship between the critical factors and the sample projects 
must be documented in the ADS so that future appraisal 
sponsors. 
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¢ There are two alternative paths to be considered 
before picking the right project in planning an 


appraisal: 
- Alternative 1: Define the organizational unit to be 
appraised. The choice of the organizational unit will 


influence the choice of projects. 
- Alternative 2: Pick the projects to be appraised. The 


choice of projects will define the organizational unit 


to which appraisal results apply. 
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¢ The advice applies when following path 1 


¢ We'll discuss “reverse engineering” the 
organizational unit from the selected projects 


later (path 2). 


ae 
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¢ Project Life Cycle Advice 
¢ Project Visibility Advice 
¢ Project Critical Factors Advice 


¢ Project Size Advice 
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1. Project Milestones and Appraisal Events: 


- Avoid choosing a project for an appraisal if the date 
for any appraisal event that requires the project's 
active participation would occur before a major 
product release or milestone. 


- Advice Rationale: The reason the project exists is to 
deliver its product(s). The project’s progress 
towards delivering the product(s) is measured by 
reaching its milestones. Anything that interferes 
with releasing product(s) or meeting milestones will 
either: 
¢ decrease the project’s likelihood of success or 
¢ be ignored by the project. 
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2. Projects Early in Their Life Cycle: 
- Avoid appraising a project if the project is early in 
its life cycle. 
- Advice Rationale: Projects that are early in their life 
cycle are likely to have: 
¢ Immature artifacts, 
¢ Sparse artifacts, 


¢ Staff that, as whole, may be uncomfortable with the project, 
the processes, or each other. 
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3. Projects Late in Their Life Cycle: 
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- Avoid appraising a project if the project will end before or soon 
after the appraisal(s) is complete. 


- Advice Rationale: Any weaknesses discovered in a project that 
is late in its life cycle will give the project limited or no 
opportunities to fix the weaknesses. Consequently: 
¢ Future appraisals (e.g.,a SCAMPI A following a SCAMPI B) using 

this project will find the same weaknesses as the current appraisal. 
¢ The project itself won’t benefit from the appraisal’s findings 
because the project may not be able to change or have too much 


invested in its existing process implementation to warrant changes 
(other projects may benefit, of course). 


Ibs: 
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1. Important Projects: 


Avoid appraising projects that are highly visible or important. 
Advice rationale: An important or highly visible piclees may 
have too many distractions or pressures that will limit its 


ability to cooperate in preparing for an appraisal or limit the 
benefits it receives from appraisal results. 


2. ‘Trivial Projects: 


il 


Avoid appraising projects that are of trivial or marginal 
importance to the organization. 


Advice rationale: 


¢ A trivial project may be cancelled, terminating its participation in 
an appraisal. 
¢ A trivial project may be viewed by the organization as not 


providing relevant process insights for other, more significant 
projects. 
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¢ Projects with Large Staff Sizes: 


- If project staff size is not a factor that influences 
processes in the organization, then select at least one 
large staff-sized project for the appraisal. 

- Advice Rationale: SCAMPI 1.2 requires that the 
appraisal population and project percentage must be 
documented. Larger projects will make the appraisal 
population percentage increase rapidly. Larger 
projects are also better able to accommodate the work 
required to prepare for an appraisal. 


¢ Hint: the SEI is looking for higher population and project 
percentages! 


= 17 
= S=—. CarnegieMellon 


=—— Software Engineering Institute 
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Before We Get Started: A Little Necessary 
Math (1) 


¢ Balanced designs are selections of projects for 


appraisals where there are an equal number of 
projects selected for each combination of critical 


factors. 
¢ Orthogonal designs are selections of projects for 
an appraisal so that the critical factors balance 
out across the projects being appraised. 
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¢ Experimental resolution helps us to understand the degree of our 
“known unknowns” in an appraisal. 


- Resolution I = we gain no insight from an appraisal 


—~ Resolution II = we cannot tell the difference between the influence of 
main critical factor effects (why bother?) 


- Resolution III = Main factor effects can’t be distinguished from two- 
factor interactions. 

° e.g., If the customer type, project size, and project type are all thought to 
influence process institutionalization, then a resolution II] experiment can’t 
tell the difference between the influence of customer type vs. project size 
and type combined. 

- Resolution IV = No main factor effects get confused with two-factor 
interactions, but two-factor interactions can’t be distinguished from 
each other. 


- Resolution V = No main effect or two-factor interaction is confused 
with any other main effect or two-factor interaction, but two-factor 
interactions can’t be distinguished from three-factor interactions. 
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¢ We will use this technique several times in the 
subsequent discussion. 


¢ Create a table in Microsoft Excel. 


- The column headings will be the names of the factors 
that influence process institutionalization in the 
organization being appraised. 


- The row headings will be the names of the projects 
that are participating in the appraisal. 
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¢ Factors that influence processes in the projects: 


- Project customer: commercial or government 
- Project location: domestic or international 


- Project type: development or maintenance 


¢ Projects being considered: 
- Project 1: government, domestic, maintenance 
- Project 2: government, domestic, development 
- Project 3: commercial, international, development 


- Project 4: commercial, domestic, development 
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¢ Fill in the table with the factor levels for each 
project 
- Consistently convert factor levels to values of “1” and 
“1 
- For each project row, fillin a “1” or “-1” depending 
on how the project is characterized against the 
corresponding factor column. 


Constructing A Project Factors Table: Step 2 
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Project 4 
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¢ Add additional columns to the table 
representing all possible combinations of factors 


Constructing A Project Factors Table: Step 3 


- Examples: 
¢ If there are two factors (Fl and F2), add a column to the table 
and label it “F1*F2”. 
¢ If there are three factors (F1, F2, and F3), add columns to the 
table labeled “F1*F2”, “F1*F3”", “F2*F3", “FI*F27F3 
- Compute the values for the cells in the additional 
columns by multiplying the values of the rows 
together according to the column headings. 


AS 
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Customer* 

. Customer* | Customer* | Location* nae 

Customer | Location | Type Location 
Location Type Type Type 


= 26 
= S=—. CarnegieMellon 


=—— Software Engineering Institute 


ZL tivsemion Project Critical Factors Advice (1) 


| Institute 


1. Higher Resolution Appraisals: 

- Select projects for the appraisal in sufficient 
numbers and diversity to get a resolution III or 
better appraisal. 

- Advice Rationale: Resolution I and II appraisals 
can't distinguish between the effect of main factors 
on process institutionalization. Resolution I and I 
appraisals yield no information. 
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¢ The organization being appraised has two factors that are thought to 


influence process institutionalization: 


- project location (California or India) and 
- project customer (US Navy or insurance companies.) 
- Project #1 is in California that has the US Navy as a customer and 
28 


¢ The appraisal looks at two projects: 

- Project #2 is in India that has Mega Big Insurance as a customer 

¢ Project #1 has some weaknesses in requirements management and 

some strengths in technical solution. 
¢ Project #2 has some weaknesses in technical solution and some 
strengths in requirements management. 

¢ What is the right conclusion about this organization’s requirements 

management and technical solution process implementations? 


¢ Answer: we can't draw any fact-based conclusions! 
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2. Critical factors versus number of projects : 


- Choose projects that maximize the number of factor 
settings in the fewest number of projects. 


- Advice Rationale: One of the principal appraisal 
cost drivers is the number of projects involved in the 
appraisal. Maximizing the number of factor settings 
covered in the fewest number of projects will 
maximize the information gained from the projects, 
and hence the coverage of the organizational unit, 
while minimizing cost. 


= 30 
= S=—. CarnegieMellon 


=—— Software Engineering Institute 


Business 


| hee Constructing A Project Factors Table 
= (Continued from Earlier Discussion) 


¢ Look for patterns or “unbalanced-ness” or “non- 
orthogonality” in the table that indicates 
potential problems: 


- One factor never varies or only varies when all other 
factors are set to particular values. 


- Two factors are always “low” or “high” in tandem 
with each other across the projects. 
¢ Indicated by one of the product columns always being the 
same sign 
- One of the main columns (with a single factor 
heading) is the same as one of the multiple column 
columns 
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Factor 2 and 3 are always the opposite of 
each other. We never learn anything about 


when they are both “high” or “low” at the 
same time. 
32 
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Factors 2 and 3 are is always the same high 
or low setting, So we learn nothing about their 


separate influences on the processes. 
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If none of the appraised sample projects includes a particular 

setting for a particular factor, then the appraisal can’t have any 

findings with respect to that factor setting. 

¢ Corollary: If your organization includes projects with a factor set 
to a particular value, then one of the sample projects must also 
include that factor with that value. 

Advice Rationale: As an example, if an organizational unit 

includes commercial and government projects, then this factor 

influences process implementation. An appraisal that includes 

only government projects could not apply to the organization’s 

commercial projects. 
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P3|1|-1|\1]| 1 | 1 
Pa] a|4|\1/| 1 | 2 | 


Factor 3 never varies—none of the appraised 
projects is telling us anything about the 
influence of factor 3 on process 
institutionalization. (This pattern is non- 


orthogonal!) 
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If the appraisal is considering three or more factors, perform “factor 
screening” SCAMPI B or C appraisal(s) before the SCAMPI A in order 
to discover and discard unimportant factors. 

Use blocking techniques to divide one large appraisal into several 
smaller appraisals. 


Advice Rationale: A full-factor appraisal that includes three factors 
each with only 2 settings each would need to examine 8 projects. 
(More factors would require even more projects.) The number of 
projects to be appraised in a single appraisal could be reduced by 
conducting lower-cost SCAMPI B or C appraisals to determine if some 
of the factors are inconsequential with respect to process 
implementation. (Alternately, use blocking techniques.) 


Example: Suppose an organization includes commercial and 
government projects. A SCAMPI C appraisal that considered 
government and commercial projects (with the other factors allowed 
to vary) and found that the customer type yielded the same process 
strengths and weaknesses could be used to conclude that the 
“customer type” factor may be eliminated from consideration for 
future appraisals. 
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¢ Obeying all of the rules of the SCAMPI MDD 1.2 


may be challenging. 
- Note: in designing experiments for engineering, 


science, medicine, social sciences, etc. , the rest of the 

world does everything described in the MDD and 
more to reach acceptable results 

¢ One way of reducing the challenge of the MDD 

is to pick the projects to be appraised and then 


compute out the organizational unit to which 
the appraisal would apply 
37 
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¢ Three SCAMPI MDD 1.2 rules drive the reverse 
engineering: 
¢ We must document_the number of people in the appraisal’s 


organizational scope divided by the number of people in the 
organizational unit (x100) 


¢ We must document the number of projects in the appraisal’s 
organizational scope divided by the number of projects in 
the organizational unit (x100) 


¢ We must understand and identify critical factors that 
influence implementation of practices in projects and 
functions within the organizational unit. 
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L_Invormein Reverse Engineering Assumptions 
¢ In documenting the appraisal, we assume that: 


- Higher population and project percentages are 
desirable versus lower percentages. 


- Any factor setting that is NOT covered in the 
appraisal cannot be attributed to the organization. 
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a. Write down a critical factors table, as discussed earlier, for the 
projects in the appraisal. 


b. Select the “maximum desired organizational unit” to be represented 
with the appraisal. 


c. Prune the “maximum desired” OU down to a “maximum possible” 
OU by checking the critical factors table for combinations of factor 
settings that are never examined — projects that contain these factor 
setting combinations must be excluded from the organizational 
unit. 

- For instance, if “customer type” could be “state government”, “federal 
government”, or “commercial”, but the critical factors table includes 
only settings of “state government” or “federal government” then the 
maximum possible organizational unit CANNOT include 
“commercial” projects. 

- Note: there is a procedure to compute the organization from the factor 
settings. However, the procedure involves solving a set of linear 
one modulo the number of factors and is too involved to discuss 

ere. 
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a. Determine the population and Prolet ercentages that 
you believe are acceptable for a SCAMPI A appraisal. 


b. Given the maximum organization determined by the 
factors coverage from step 1, compute the population 
and project percentages for the actual appraised 
projects compared to this maximum organization. 


- If the computed population and project percentages from step 
2b are greater or equal to the limits in step 2a, then the 
organizational unit may be set to the maximum organization 
for the appraisal. 

- Ifthe computed population and project percentages from ste 
2b are less than sige imits in step 2a, then reduce the scope o 
the maximum organization until the actual population and 
paces percentages are greater than or equal to the limits set in 

a. The organizational unit for the appraisal is this reduced 
maximum organization. 
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¢ We want to pick the right projects for an appraisal in 
order to: 

~ Maximize the information gained about an organization's 
process implementation, 
- Minimize the cost and risk of the appraisal, and 

- Produce results that are credible to the larger process 

improvement community. 


mathematical techniques such as design of experiments. 


¢ We've discussed a number of approaches for achieving 
these goals based on our appraisal experiences and using 
- The methods we've discussed require tools no more complex 
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¢ Bob Moore, Business Transformation Institute, Inc. 


—- rlmoore@biztransform.net 
¢ Will Hayes, Software Engineering Insitute 


—- wh@sei.cmu.edu 
¢ Additional information with respect to MS Excel 
computations and background material may be 


found at www.biztransform.net under 


“Publications’ . 
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Background 


*Z = Many organizations have implemented the Capability Maturity 
ey Model Integrated (CMMI) 


1 = Although they have achieved their desired maturity level and 
improvement goals, some organizations have seen little or no 
financial benefits 


What are the underlying principles of CMMI as they relate to 
productivity, predictability, and speed? 


What is the return on investment? 


What are the timelines for realizing these benefits? 


CMM®, CMMI® are registered trademarks of Carnegie Mellon University 
NORTHROP GRUMMAN 


2 Copyright 2005 Northrop Grumman Corporation 


= How Do Mature Processes Help? 


= Knox Cost of Quality model 
= Industry ROI data for CMMI 
= Why ROI data doesn’t always help 


= Extracting strategic value from CMMI 
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Projects Have Historically Suffered from 
Mistakes 


People-Related Mistakes 

1. Undermined motivation 

2. Weak personnel 

3. Uncontrolled problem 
employees 

4. Heroics 

5. Adding people to a late 
project 

6. Noisy, crowded offices 

7. Friction between developers 
and customers 

8. Unrealistic expectations 

9. Lack of effective project 
sponsorship 

10. Lack of stakeholder buy-in 
11. Lack of user input 

12. Politics placed over 
substance 

13. Wishful thinking 


Process-Related Mistakes 
14. Overly optimistic schedules 
15. Insufficient Risk 
Management 

16. Contractor failure Insufficient 
planning 

17. Abandonment of planning 
under pressure 

18. Wasted time during the 
fuzzy front end 

19. Shortchanged upstream 
activities 

20. Inadequate design 

21. Shortchanged quality 
assurance 

22. Insufficient management 
controls 

23. Premature or too frequent 
convergence 

25. Omitting necessary tasks 
from estimates 

26. Planning to catch up later 
27. Code-like-hell programming 


Reference: Steve McConnell, Rapid Development 


Product-Related Mistakes 
28. Requirements gold-plating 
29. Feature creep 

30. Developer gold-plating 
31. Push me, pull me 
negotiation 

32. Research-oriented 
development 


Technology-Related Mistakes 
33. Silver-bullet syndrome 

34. Overestimated savings from 
new tools or methods 

35. Switching tools in the middle 
of a project 

36. Lack of automated 
source-code control 


Standish Group, 2003 
survey of 13,000 projects 


¢ 34% successes 
e 15% failures 
e 51% overruns 


NORTHROP GRUMMAN 
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Many Approaches to Solving the Problem 


= Which weaknesses are causing my problems? 
= Which strengths may mitigate my problems? 


= Which improvement investments offer the best return? 


Business 


Environment Management 


Structure 


age 


Product Methods 


a Tools 


One solution! 


Technology 


NORTHROP GRUMMAN 
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Approaches to Process I mprovement 


Data-Driven (e.g., Six Sigma, Lean) Model-Driven (e.g., CMM, CMMI) 


Y 


= Clarify what your customer = Determine the industry best 
wants (Voice of Customer) practice 
= Critical to Quality (CTQs) = Benchmarking, models 
= Determine what your processes = Compare your current practices 
can do (Voice of Process) to the model 
= Statistical Process Control = Appraisal, education 
= Identify and prioritize = Identify and prioritize 
improvement opportunities improvement opportunities 
= Causal analysis of data = Implementation 


= Determine where your = Institutionalization 


customers/competitors are = Look for ways to optimize the 
going (Voice of Business) processes 


= Design for Six Sigma 
NORTHROP GRUMMAN 


| How Do Mature Processes Help? 


Process maturity gets at one J. Herbsleb and D. Zubrow, 
source of the problem, e.g., “Software Process Improvement: 
= Are we using proven An Analysis of Assessment Data 


industry practices? and Outcomes” 
= Does the staff have the = 13 organizations 
resources needed to = ROI of 4:1 to 9:1 
execute the process? = Improved quality, error rates, 
= Is the organization providing time to market, productivity 


effective project support? 


= The main benefits typically 
seen are: 


= Improved predictability of R. Dion, “Process Improvement 
project budgets and and the Corporate Balance Sheet” 


schedules 

= Improved management = ROI of 7.7:1: Reduced re-work, 
awareness of problems improved quality | 

= Reduced re-work, which = Two-fold increase in 
improves predictability, cost, productivity 


and schedule 


NORTHROP GRUMMAN 
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The Knox Cost of Quality Model 


= Extension of the Cost of Quality model used in manufacturing 


Category Typical Costs for Software 


_ 


Prevention 


Internal 
failures 


Non- 
conformance 


External 
| failures 


“Knox’s Theoretical Model for Cost of Software Quality,” Digital 


Discovering the 
condition of the 
product 


Efforts to 
ensure product 
quality 


Quality failures 
detected prior 
to product 
shipment 


Quality failures 
detected after 
product 
shipment 


Technical Journal, vol.5, No. 4., Fall 1993, Stephen T. Knox. 


Testing and associated 
activities, product quality 
audits 


SQA administration, 
inspections, process 
improvements, metrics 
collection and analysis 


Defect management, rework, 
retesting 


Technical support, complaint 
investigation, defect 
notification 


NORTHROP GRUMMAN 
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Knox Model - Theoretical Benefits 


-©- Prevention-#— Appraisal -& Int Failure -e— Ext Failure-m- TCoSQ 
60 


= s 
ol 
(o) 


IS 
(oe) 


Cost as a Percent of Development 
NO ioe) 
io) jo) 


- 
(o) 


1 2 3 4 5 
SEI CMM Level 
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Benefits 


re ee = Numerous studies have 


oe =| a ee ee been published on the 


‘ MANAGEMENT = Concepts and Benefits of CMMI” be n efi ts of CM M l 
se mam wetenearautaanantacied AS 


" Bhutts CMMI? process impeowerment 


i = Performance Results of 

CMMI-Based Process 
i — . Improvement (CMU/SEI- 
manacemn™ — Cant Performance Results ‘ 2006-TR-004) 


Ah De ee Pe ten 
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Everts, & putlicly available conterence presentations, putashed — we Ss I e . 


Fosuers pape, and indhedual cetlaberations wi the SET 

* Pero mance 7 
it 

We Naa, se tt pre rf of net et at ce http://www.sei.cmu.edul 
‘ f based process improvement The results show that CMMI often leads to wery q 
w inp restive inpr ovement: © product quaily, project pertoimance, and = 

ee organizational perform ance, hoawewer, the mdrwidualresats presented hete Cc mM m i/res U lts htm | 
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Typical CMMI Benefits Cited in Literature 


=a 


cg 


Reduced Costs 
= 33% decrease in the average 


cost to fix a defect (Boeing) 


20% reduction in unit software 
costs (Lockheed Martin) 


Reduced cost of poor quality 
from over 45 percent to under 
30 percent over a three year 
period (Siemens) 

10% decrease in overall cost 
per maturity level (Northrop 
Grumman) 


Faster Schedules 


50% reduction in release 
turnaround time (Boeing) 

60% reduction in re-work 
following test (Boeing) 
Increase from 50% to 95% the 
number of milestones met 
(General Motors) 


= Greater Productivity 


=» 25-30% increase in 
productivity within 3 years 
(Lockheed Martin, Harris, 
Siemens) 


=" Higher Quality 
= 50% reduction of software 
defects (Lockheed Martin) 


= Customer Satisfaction 


=» 55% increase in award fees 
(Lockheed Martin) 


NORTHROP GRUMMAN 
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Why Do We Need ROI Data? 


= Management wants to invest overhead 
resources wisely 


= Similar investment decisions are often based 
on “gut feel”, not hard data — does anything else 
seem more likely to yield results? 

= Investment decisions may be more driven by balance of short- 
term performance tactics and long-term marketing strategy 

= The key question is whether you could make similar progress 
with less resources (or more/faster progress with the same 
resources doing something else) 


= Projects want to justify the investment to their customers 
= Difficult to convince process skeptics 
= People view the problem from their own experiences and skills 


Beware of ROI as a smokescreen for process skepticism 


NORTHROP GRUMMAN 
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Establishing ROI 


" Itis difficult to quantify the value 
of an improvement initiative 


=" How do you measure the change? 
= Multiple levels — organizational, management, engineering, 
support 
= Multiple causes — awareness, knowledge, infrastructure 
= Short-term vs. long-term — Hawthorne effect 


=" How do you measure the investment? 
= What would we have done instead? 


= How do you determine the value of the measured change? 
= Increased predictability — what’s the value? 
= Increased productivity — who gets the benefit? 
= Better competitive position — how measured? 
=" Time-frame 


See also: S. Sheard and C.L. Miller , "The Shangri-La of ROI," NORTHROP GRUMMAN 
Software Productivity Consortium, 2000 
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Where the Problem Sometimes Arises 


= Some organizations are driven to achieve a maturity level only 
for it’s marketing value 


Improvement goals are ! No one takes the improvement 
realistically (“Level 5 in effort seriously 


Only some of the proje Personnel perceive CMMI as 
in the improvement effo more expensive 


Only some of the projec Others don’t implement; 


appraised perceived as “done for show” 


Insufficient resources ( People don’t learn the new 
QA, metrics, consultan behaviors or become proficient 


Management doesn’t ent Perceived as “done for show”; 
process benefits are not realized 


NORTHROP GRUMMAN 
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Extracting Strategic Value from CMMI 


rd 
es / Project Performance Organizational Performance 
z 
J Quality/Rework Institutionalization 
e 
NORTHROP GRUMMAN 
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“Project Implementation Strategies in the CMMI,” R. Hefner, Wednesday, 3:30 


jy Project Performance 


= Project performance = Identifies the elements of good 
problems often arise planning 
because of incomplete or = Proven engineering processes 
unrealistic planning oe 
.£ ee = Estimates based on historical 
orgotten activities data, using these processes 
= Unconscious decisions ce ened 
m -ovtimisti " en cost/schedule pressure 
Os Guo ollintsile etl ecs arises, CMMI practices track and 
= When cost/schedule correct 
pressure arises, people = Reactive (L2) 
abandon the plans, leading = Proactive, risk management 
to more problems (L3) 
= Individual judgment versus = Quantitative management (L4) 


best use of resources 


=" QA, management ensures 
processes/plans are followed 


= Train project managers on how to use the tools (estimation, 
earned value, risk management) 

= Project managers (not organizational staff) must be responsible 
for implementing the improved processes 

= Demand realistic, data-driven estimates 


NORTHROP GRUMMAN 
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| Organizational Performance 


= Each project’s processes are = Standard organizational 
unique process, tailored to fit each 
= Personnel must re-learn project 
with each project = Can be documented, trained, 
= Difficulty moving people supported by templates 
from project to project = Over time, people learn the 
= Historical data of little use in process 


estimation 
= Common processes/measures 


= No way to compare project- allow better use of historical 
to-project data 
= Which process was best? = Calibrate cost estimation 


= What did we learn? models 
= Project to project comparisons 


= Over time, the organization can 
optimize the process 


= Develop an organizational process(es) which fits the full range of 
your projects (small/large, all life cycles and project types) 
=" Capture and use historical data (measurement repository) 


=" Capture and share project documents (process asset library) 
NORTHROP GRUMMAN 
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| Rework/ Quality 


18 


= Focus on “fasterand _ = A disciplined engineering and 
cheaper” leads to skipping of management process 


essential steps = Do it right the first time 


* Key steps are not obvious, = CMMI identifies the essential 
often counter intuitive steps 


" Fixing latent defects often = Peer reviews find defects early, 


accounts for 30-40% of where it is cost effective to fix 
project cost fehl 


= The cost of defects (rework) 


= Requirements, designs, code, 
is seldom measured 


plans, etc. 


= Often more efficient and 
effective than testing 

= Many types (Fagan 
inspections, walkthroughs, 
desk checks, etc.) 


=" Focus on eliminating defects, not on faster and cheaper 

= Measure the cost of finding and fixing defects 

= Invest time in learning different methods of peer review and when 
each Is effective 


NORTHROP GRUMMAN 
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I nstitutionalization 


= Some improvement efforts = Short-term investment for long- 
focus on quick fixes term gain 
= Driven by yearly budget = Initial investment in the cost of 


cycles change, learning curve, new 
= Expectation that results will overhead structures 
be immediate = Long-term benefits in increased 
productivity 


= Itis tempting to reduce 


overhead to reduce cost = Organizational infrastructure 
= Training exists to support the policies 
= Staff support to projects and process 
= Use of outside process = Measurement repositories 


experts 


=" Expect 18-24 months before benefits begin to be realized 
= Senior management must demand that everyone follow the new 
processes 


" QA can be the organization’s strongest tool -— if they are focused! 
NORTHROP GRUMMAN 


19 “Sustaining CMMI Compliance,” R. Hefner, Thursday, 10:15 
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Lessons Learned 


, = Process improvement means changing the process 
? = More important to learn the new behaviors than to “go through 
the motions” 


= Resistance often comes from fear of failure 
= Walk the talk -- management at all levels must communicate the 
need for continuous improvement 
= Focus on learning from your mistakes and getting better 
= Training and assistance helps people in trying new processes 


= Six Sigma is a strong enabler for process improvement 
= Focus on data, measurement systems, process improvement 


= Tying improvements to business goals 
= Allows the projects and organization to optimize the CMMI 
practices for maximum customer benefit 


NORTHROP GRUMMAN 
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—— The Traditional View of 
proces »ssMaX 


nt ascot TOUT Measurement & Analysis 


Establish Specify Data Specify 
Objectives Procedures Procedures 


I 
I 
J I 
Measurement Monknrac Collection Analysis : 
I 
I 


Measurement 


Se a ee 
| 
I ; 

store 
Communicate Data & ge dg ac 
Results Results | 
| | 


Zubrow, David. Measurement in a Process Framework. Process and Methodology Seminar, 


Toronto, Canada, 2003. http://www.sei.cmu.edu/sema/pdf/zubrow_framework.pdf 
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FT cecah 


cessMax. Problems with the Traditional View 


e TOO many complex activities must be 
ofolan) e)(=)o1em elove)com-lal\vmanlcraKielacianiclalarsl are! 
analysis can be performed 


cum DY-Ve- exe) |(=1e1 (eam ismalelmaiccie|e-ccre mnie 
(e) folerotoiom ol -) a(e)enat-lales 


¢« Concept of “measurement personnel with 
oi fe) okey- ge Koma tomlaloiilel(-sa)mr-lalemlalcliccreunic 
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Measurement Challenges 


cam = || Aa) ata lecme) alc) ce )lutcw-lalem-10c0) eo e)ce)alcmel-ic-mee)|(-rellele 
e) celer-rel 0] cers 


e Deal with the large number of measures required 
by the CMMI in a consistent manner 


Sian wl ats e)(smerelsyce)anlyAsiulelame)maalststielasmelsvilalinelamsiare 
reporting 


e Integrate data collection with day-to-day work 


e Enable real-time, fact-based management insight 
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oo The processMax View of 


‘Ocess 
processMax Measurement & Analysis 


Pre-performed by processMax 


Align Measurement Activities 


Specif 

Establish Specify Rete” Specify 
Measurement easel Collection Analysis 
Objectives Procedures 


— 


Measurement 
Repository 


t 
Available for : 
Reassignment °opmmcate 


Results 


Data & 
Results 
I 


Automated by processMax 
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processMax. processMax Measurement Approach 


HE PRO OL 


¢ Data is automatically captured at the source as user follows 
process steps 
— Timely 
— Accurate 
— Not subject to “revision” 
— No burden on users or measurement personnel 


e As required or on a scheduled basis, a user initiates report 
generation 


e Report generation accesses the Measurement Repository 
to retrieve relevant base measure data, generates and 
stores the report in the project repository 


e Any other user can browse the stored report, with drill-down 
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processMax 


pcesenex Organization-Project Structure 


orgs rlalycciicelarclm cel arel| 


Organization’s Standard 
Project Process 
Defined Process 
Project Management Roles 
Project Technical Roles 
Templates 

¢ Guidelines 

¢ Criteria 

Methods 

Forms 


. 
oe? 
.* 
Po 
. 
. 
. 
. 
. 
.* 
- 


z alsitelalite\icm Ol ce F-Valr4:lalel aie) 
i Standard Project Process 


¥ =o) dp atalin mrarsve | orale 
Wi texevarslalcsiaats 


f 


Project 1’s personnel use this website Project N’s personnel use this website 
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processMax 


Report Formats and Storage 


Measurement Repository 


Data Collection Services 


Standards-Based Measurement 


ISO/IEC 15939, 


Software Engineering — Software 
Measurement Process 


— Robust measurement process standard 
consistent with CMMI requirements 


aoe bol @) A) =i Ou \V(atatsielasiaatcvalmialce)aaatsiulelam\y(ele(s) 
(MIM) provides structure that links information 
als{s\Okm KOM as) (\VsclAlMsalililstomrslalOmualsiimrsluealele|ots 


©2006 pragma SYSTEMS CORPORATION Page 9 www.pragmasystems.com 


— ~¢ 


processMax. Key Relationships in ISO/IEC MIM 


THE PROCESS SOLUTION 


il a(=meo)Ul coro) aaoe)mual= 


Information Needs measurement process that 


Faliksyilesomd ale lalce)anatclice)amaletsvets 


7Nfofe)aidalanmie)mece)inle)ialiale Mantexctsie ness 


and decision criteria 


oe 


Variable assigned a value by 
applying the measurement function 
to two or more base measures 


PN [ofo) sidalanmceymecyanie)ialiale mii enele 
more base measures 


=> 0) felare tela ccatcUlarem aloe lUrellilecliN cm ial elaine lelamia 
Liat lave |terc\co) mm (Oat= mi nicelanntele)amarcr-re lcm Amiel 
language of the measurement users 


Variable assigned a value by applying the analysis 
model to base and/or derived measures 


Variable assigned a value by 
applying the method to one attribute 


©) of=}¢-\ite)akomantele) e)iarepraua 
attribute to a scale 


Property relevant to 


HTalce)anateuite)amalsxexe ls) © ISO/IEC 2002 
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, — CMMI Level 3 for Software and Systems 


processMax. Engineering 


THE PROCESS SOLUTION 


e Measurement Objectives — 19 


— Approximately equivalent to the Purpose section of each Process Area 
(answers the question “Why are we measuring these particular items?”); 
associated with one or more Information Needs 


e Information Needs — 24 


— Correspond approximately to PMC SP 1.1 and GP 2.8 of each Process Area; 
Felsysolerr=1cciemVVisamelalcme) auenle)com [alel(er-lie) 6s 


e Indicators — 83 
— Trends or snapshots relying on Derived Measures and/or Base Measures 
e Derived Measures — 115 
— Values that are functions of Base Measures and/or other Derived Measures 


e Base Measures — 156 


momen /alie)aagt-tecelame(-sa\cromice)anur-Msyiale](cmr-ling|o]bicomm |alel-)e\-1ale(-alme)melial>)m@st-ki> 
Measures 
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Why Earned Value Management? 


Enables determination of schedule 
Welatelalet= 


Enables determination of “true cost 
variance”, not distorted by schedule 
oleyaco)danrelalers 


=) Not possible with traditional 
budget vs. actual cost tracking 
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Basic Concepts of Earned Value 


Management 
e Plan all work for the program 


e Integrate work scope, schedule, and cost 
objectives into a baseline plan 


in ©) 0) [=1eithVAc) \Vare Koysiosstourelexexe) an) e)iisialpalcialis 


e Analyze significant variance from the plan 
and forecast impacts 


on wa £0) VA(0(omer~lk- Wie) anal (elalcimulcaViclmanrelarcle(c)aelovall 
oltolKse)amanrel.diaremelaremianle)icieatciaiccvilee 


Earned Value Management Systems, ANSI/EIA-748-A-1998. 
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Earned Value Management CMMI 


Moye) Grint, 


male) (210) wate lavaliale 


SG 1 Establish Estimates 
Sts} ilgarclcctsme) m0) ce)(-ce1m e)t-Valalialem er-leolaalcitclesm=|Romots)r-1e)| alors) 
relate maatsljaltslialcren 

SG 2 Develop a Project Plan 


PANS ¢) 0) (=Yoim 0)(-lamismcscir-le)|syarcre mre lal mm pare llalralialsye m= Komaglcom Or-Ksyiom (0) 
gare larcle|ialemaatcm @)4e)(-1e1m 


make) (=e1m \V/(e)alice)diale msl alemene) alice) 


SG 1 Monitor Project Against Plan 
PN oi i0r-1m el=1ai0)dpatelalercm=|a(em 6) cele |cossssme) mi alom 0) 40) (-1e1 m= lac 
ante) aiite)ecremrelereljats)meatom 6) (lap 
SG 2 Manage Corrective Action to Closure 
Corrective actions are managed to closure when the project’s 
performance or results deviate significantly from the plan. 
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processMax Strong CMMI-EVMS Relationships 


THE PROCESS SOLUTION 


CMMI EVMS Guideline from EIA-748-A, Earned Value Management Systems, Jan. 
Process Area | 2902) [Earned Value Management Maturity Model® Goal] 


Project 2.1a Work Breakdown Structure (WBS) [L2, Organizational, Goal 1] 
. 2.1b Organization structure [L2, Organizational, Goal 2] 
Planning 2.2a Schedule of authorized work [L3, Planning, Goal 1] 
2.2b Progress indicators [L3, Planning, Goal 2] 
2.2c,d Control account budget baseline [L2, Planning, Goals 1; L3, Planning, Goal 3] 
2.2e Work packages [L2, Planning, Goal 2] 


Project 2.4a,b Schedule/cost variance analysis [L2, Analysis, Goal 1] 
; : 2.4c Indirect cost variance [Level 3, Analysis, Goal 3] 
Monitoring 2.4d Element summary [Level 3, Analysis, Goal 4] 
2.4e Managerial actions [Level 2, Analysis, Goal 3] 
and Control 2.4f Estimate at completion [Level 3, Analysis, Goal 5] 


Integrated 2.1c Integration of plan, schedule, budget [L3, Organizational, Goal 1] 
: 2.1e Integrate WBS and org structure [L3, Organizational, Goal 3] 
Project 2.2a Identification of task dependencies [L3, Planning, Goal 1] 
2.2d Budgets for authorized work [L3, Planning, Goal 3] 
Management 2.4 a-f (see Project Monitoring and Control) 
2.5a Incorporation of changes into budget and schedule [L2, Revisions, Goal 1] 
2.5e Changes to performance measurement baseline [L2, Revisions, Goal 4] 


Measurement 2.2b Progress indicators [L3, Planning, Goal 2] 
and Analysis 


Solomon, Paul. Using CMMI to Improve Earned Value Management, 
CMU/SE]I-2002-TNO16 
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ssMox. Essence of Earned Value Management 


e Assign a target value to each scheduled 
Ve) a @rod(c) paloialin (almrslamral@)e)ce)e)ar-ltcm [oc] me) 
detail) 


e As work elements are completed, their target 
values are “earned” 


e Work progress Is quantified by earned value, 
against which we can measure: 
— What was spent to perform the work 


— What was scheduled to have been 
accomplished 
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Why Do CMMI and EVM Together? 


Ensure that processes are integrated 


al /anliarclicm @)celercrstomere)a)i|(eikcmr-lale 
KetolUlalerelalelcrs 


e Reduce rework that is inevitable from 
sore U[ovalireVimlan) e)icipatcvaleelareya’ 


an =4 ale) (-mro ere laniagle)amsir-lanel eel e)e)cey-lea 
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. Bringing It All Together for 


roc cessMax 
process Earned Value 


e Measurement Objective 
meas =101S10 | (oneal com 0) @)( -101mm Kom ol-1| ale mm Ol-1a(@)anatcremulivallamliecm e)r-lalaliarem er-leclagclicles 
(PMC) 
e Information Need 
— Monitor schedule, effort, costs, resources, and estimates for work 
product and task attributes (PMC SP 1.1) 
male | (ere tne) 
— Trend of Cumulative Earned Value (PMC SP 1.1, Subpractices 1 & 2) 


e Derived Measures (top-level) 


— Planned Value, Earned Value, Actual Cost, Cost Variance, Cost 
Variance Percentage, Cost Performance Index, Schedule Variance, 
Schedule Variance Percentage, Schedule Performance Index, 
mts} patalccle i @xe)<) m= | m@xe)anle)(-ilela 


©2006 pragma NSN ESPUDIVISN Ox0)5110) 7,40 CONN) Page 18 www.pragmasystems.com 


— ~¢ 


processMax Derived Measure Derivation 


THE PROCESS SOLUTION 


mts laalsvemvasl (els 


sum (Planned Cost to Date per Task where Task is Level of Effort, 
Earned Value per Task Where Task is not Level of Effort) 


Earned Value per Task Where Task is not Level of Effort 


multiply (Percent Complete per Task where Task is not Level of Effort, 
Planned Cost per Task where Task is not Level of Effort and 
Actual Start Date is less than Date of Report) 


Planned Cost per Task where Task is not Level of Effort and Actual Start 
Date is less than Date of Report 
sum (Planned Other Direct Costs per Task, 
Planned Labor Cost per Task where Task is not Level of Effort and 
Actual Start Date is less than Date of Report) 
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x. Trend of Cumulative Earned Value Report 
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DOD Weapon Systems Acquisition 
Continues To Be High Risk Area 


While DOD’s acquisition process has produced the best weapons in the 
world, it also yields undesirable consequences in weapon system 
programs - cost increases, schedule delays, and performance shortfalls 


Problems occur because weapon programs do not capture early on the 
requisite knowledge that is needed to effectively manage risks 


Programs move forward with unrealistic cost and schedules estimates, 
lack clearly defined and stable requirements, use immature technologies, 
and fail to solidify design and manufacturing processes at appropriate 
junctures in development 


As aresult, programs require more resources than planned, the buying 
power of the defense dollar is reduced, and funds are not available for 
other competing needs 


Paul Francis, Director i 

Acquisition and Sourcing Management — G A O 
U.S. Government Accountability Office SE — Accountability + integrity + Reliability 
May 18, 2004 
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Problem Statement 


A statement of organizational process maturity or capability level 
does not guarantee performance to that same level of proficiency 
on an individual project 


Most DoD contractors claim high maturity/capability levels, yet 
from the perspective of the acquirer, systems engineering and 
program management practices are severely lacking 


Teaming arrangements further cloud the issue of process 
execution and proficiency 


Associated problems may not be evident until significant cost, 
schedule, or performance objectives have been missed at a late 
point in the program where corrective actions are very costly 
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High Maturity 
Organizations 


Maturity Levels are 
indicators of organizational 
potential performance. 


High Maturity 
Projects 


XYZ Corp. 


Division 
N 


They describe how the next 
project may perform based 
on a sampling of existing 
projects. 


Maturity Levels reside at the 
organizational level and are 
not an indication of how an 
individual project /s 
performing. 
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MLs apply re but you \ 
HERE project Is 

based upon HERE or HERE 
appraisals of 
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Standard CMMI Appraisal Method for 
Process Improvement (SCAMPI) 


Benchmarking and 


Baselining 
re Tailorable Assessment Class A 
© “Mini” Appraisal Method 
ra as executed 
oO : igs 
x Maturity or Capability 
oO Class B ratings 
2 Quick Look Methods 
ce “Intentions” for execution 
= No ratings 
fe) Class C 
O 


Methods 


No ratings 


Cost/Duration 
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PIER Ground Rules 


e Use the process model —- CMMI 
e Interview questions based on model 
¢ Appraisal of process performance and adherence 


¢ Focus on risk assessment - risks associated with 
process performance, adherence, and capability 


¢ Observe strict confidentiality 


e Results not attributable to individuals or interview 
groups 


¢ Approach SCAMPI collaboratively 


¢e Results in actionable findings by Program Office 
and/or Contractor 
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Mission Planning Contracting Structure 


i 


Delivery Orders 


Contractor DO 
#1 S: 


#2 
Contractor 
#3 
Contractor 
#4 
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How PIER was Employed for MPEC 


e Used appraisals to baseline compliance with CMMI requirement 
¢ demonstrate capability or 
¢ develop findings which yield improvement opportunities 


¢ One project would be selected from each contractor as a 
‘representative’ of all Mission Planning projects for that contractor 


¢ Appraisal findings would be used to assess progress toward 
meeting this the 24 month CMMI requirement 
e Results to be factored into future DO competitions 


¢ Appraisal finding to be used for contract monitoring to identify 
areas of risk in project execution 


¢ Appraisal findings resulted in request for Process Improvement 
Plans from the contractors 
¢ Way ahead to fix findings in representative project 


¢ Way ahead to fix similar known deficiencies in all Mission Planning 
work 
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PIER Results have changed the way 
ie Program Office does business 


Contractor Process Improvement Plans and status are tied to contractor 
award fee 


¢e RFP language has been modified to better reflect the program office 
desire for CMMI compliance across development teams - including 
subcontractors 


¢ Based on input from appraisals, the program office is 
reviewing/modifying the standard CDRL list to get best ROI 


e Business and technical rules for MP developers have been modified 


¢ Program Office is modifying internal processes to better take advantage 
of information provided through contractor’s standard processes (e.g. 
metrics) 


“In light of significant personnel cuts, we need to provide asmart way to 
provide a laser-like focus to our key issues. PIER reviews and EVM analysis 
provide that focus for us.” 


Steven A. Cote, GS-15, DAF 
Deputy Director, 951st Electronic Systems Group 
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4A 
BLUE Force 
Positive lessons / Benefits 


¢ Overcoming common misperceptions program offices have 
about CMMI 


- What does a maturity level mean — how Is it attained 
- Need to understand the scope of previous appraisals and 
certifications 


— Organizational unit appraised 
e More savvy consumers 


- Monitoring is more necessary than people think 
- CMMI compliance and appraisal finding resolution should be tied 
into program office business activities 
— Award Fee Criteria (powerful motivator) 
— Business and Technical Rules 
— Proposal Evaluation Criteria 
¢ Appraisal results can be used to assist program management 
- PMRs can focus on areas of concern based on findings 
e Uncovering program risks is more important than specific 


maturity levels — real time contract monitoring is key 
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Lessons Learned 


Hard lessons / Ongoing challenges 
¢ Spirit vs. letter of the model 


- Assure intent of the model is met versus using “checklist” 
mentality 


- Determine consistent approach to dealing with corporate 
“orocess lawyers” 


- Shouldn’t judge “goodness”, but you should judge 
“reasonableness” 


¢ Variability in the process itself, within the allowable scope of 
the method, can itself modify results 


- Teams take on personalities based on membership (team 
members and lead appraisers) 


— Different personalities can provide inconsistent results 
- Need to ensure that decision criteria are consistently applied 


- Having acore group as a subset of each appraisal team 


provided a consistent methodology and interpretation of 
evidence 
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e Mission Planning is reviewing options for verification of 
the November 2006 CMMI Maturity Level 3 requirement 


¢ Accept results of contractor SCAMPI As? 
- Review of detailed appraisal results 


e Conduct Program Office sponsored SCAMPI As? 
- Requires core team to re-appraise contractors 


¢ Verify completion of all process improvement 
activities outlined in approved Process Improvement 
Plan 
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Way Ahead -2 


eHow do we “maintain” the process improvement/process 
maturity focus to ensure consistent levels of 
performance/execution? 


e The Program Office is working to develop a multi-year 
plan for continual assessment/monitoring 


- Additional appraisal for all contractors (cost/benefit) 
-“Spot check” areas of known deficiency 
- Appraisals for projects with performance issues 
-Appraisals for projects that fall within another 
organizational unit than “representative” project 
¢eBeginning to analyze PIER results against product quality 
to determine if there are correlations 


¢ Could impact risk management/corrective actions in the 
future 
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Summary 


e PIERS were very positive for Mission Planning 
e Provided insight into contractor processes and 
potential program risks 
¢« Consistency is key 
e Appraisal results have become a useful source of 
data to assist in program management activities 
e PMs can focus attention on areas of the project that 
have highest risk due to inconsistent (or non-existent) 
standard processes 
¢e PIERs started as a way to do a “quick check” of 


contractor CMMI requirement compliance —- and have 
now become a powerful management tool 


Savvy consumers can utilize PIERS as part of their 
management “tool box” 
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For More Information, Contact: 


Lorraine Adams Kathy Bastien 
Member of the Technical Staff Senior Process Manager 
Software Engineering Institute BlueForce LLC 
4500 5" Avenue 1919 Commerce Drive, Suite 230 
Pittsburgh, PA 15213-3890 Hampton, VA 23666 
412-268-7777 757-788-8441 


ladams@sei.cmu.edu kbastien@blueforcellc.com 


Applying CMMI 
to the 


ia DCMA Surveillance Strategy 


2 Re ’ 
ra aE? oF 


ate ye m 


Presented by: 


John Eget 
DCMA - Pittsfield 


Nov 15, 2006 


DCMA Mission 
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<=, We Enable t he 
Bp, ein to Win 


DODD 5105.64, 9/27/2000: Mission — DCMA shall perform Contract Administration 
Services (CAS) for the DoD, other authorized federal agencies, foreign governments 
— and others as authorized. DCMA is a combat support agency of the DoD under 
the direction of USD (AT&L) performing CAS functions [AW the Federal Acquisition 
Regulation and DoD Federal Acquisition Regulation Supplement. 


DCMA Topics to be Discussed 
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¢ DCMA - Pittsfield Suppliers Time Line from CMMI Level 3 to 
CMMI Level 5 


¢ Fundamentals of using CMMI Based Performance Management 
(CBPM) in a Performance Based Management (PBM) 


environment 
¢ DCMA Pittsfield Method Description (CBPM methodology) 


¢ Measurement Application using modified Practical Software 
Measurements (PSM) structure 


¢ Surveillance Task/Action to CMMI Process Helper Guide 


¢ DCMA Pittsfield Data Analysis and Use 
3 


DCMA DCMA Pittsfield Suppliers Transition 
¢ CMM Level 3 - Prior to Nov 2003 


¢ Data recorded and analyzed using a CMM Based surveillance 
a application 
¢ CMMI Level 3 — Nov 2003 by contracted external 
assessment 
¢ Data recorded and analyzed using CMMI CBPM surveillance 
application 
¢ CMMI Level 5 — Nov 2005 by contracted external 
assessment 


¢ Increased scope of PA coverage & Continued Data recording 
and analysis using CBPM surveillance application 
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DCMA Institutionalization of a Process Change 


Initiate sponsorship and establish infrastructure 


Define Customer Outcomes (CO) and transfer to 
Memorandum of Agreement (MOA) elements 


Determine tasks/actions to be performed from MOA 
and select measurements to be collected/reported 


Establish priorities and develop measurement plan 


Initiate the measurement plan — start collecting and 
analyzing data 


Report DCMA surveillance analysis results to influence 
5 
Customer Outcomes 


DCMA CMMI Based - Surveillance Framework 
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- Step1 Plan & Prioritize Surveillance 
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' 
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Categorize Surveillance Results 
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DCMA REPORT 
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Surveillance 
Results 


Report Surveillance and 
Make Recommendations 
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DCMA Plan and Prioritize Surveillance 


Defense Contract Management Agency 


“stp 


Customer Outcomes (CO) establishes Memorandum of Agreement 
(MOA) between Program Management Office (PMO) and DCMA 


¢ Analyze identified factors to determine and prioritize Critical 
Process Areas and Practices 


¢ Develop DCMA Surveillance Plan 


MOA specified tasks/actions 
Past Performance 
Integrated Master Schedule 


_ Step1 Plan & Prioritize Surveillance | step2 Implement Surveillance 
Proces: i 


Critical Path 
Statement of Work 


DCMA _ Implement Surveillance -Step 2 _ 
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Perform Process Reviews, Product Examinations 


-lf necessary, issue Contractor Corrective Action (CAR) or Process 
Improvements (CIO) 


¢ Tasks/actions for Process Reviews and Product Examinations 
determined in Planning and Prioritization 


¢ Tailored PSM Measurement Application Process in Measurement Plan 


«Practices observed determined By DCMA CMMI pela Guide 


¢Record strengths as well as weaknesses 


DCMA Tailored Measurement Application Process 


Defense Contract Management Agency 


Collect and Rausing 
Process Data 


Define and 
Generate 
Indicators 


Software 


Derived Questio 


Issues eee 


Report 


Analyze 
Issues 
Results 
Other Program > 


Information 
Take 
‘Surveillance adjustment _ a 
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DCMA Tailoring the PSM Process 
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¢ Memorandum of Agreement (MOA) is the primary driver for 
measurement requirements to satisfy Customer Outcomes (CO) 


¢ The supplier’s process defines how the software is to be measured 
throughout the life-cycle 
¢ Reconciliation of available metrics possible 


¢ Collect, record, and analyze data to satisfy MOA derived 
tasks/actions, CMMI Process Areas/Practices (PA/P), and identified 


Software Issues/ / Customer Outcomes/MOA 
issues “ = 


¢ Apply independent analysis 


Project Health by CMMI PA 


- Use a structured approach to ensure tie, = 
analysis results are related to established _.,,.... \ 


Analysis result/Insight 


Customer Outcomes soeneroaamen | t_Y 
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DCM A recora, Classify, and Categorize Surveillance Results Step 3 
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Recording of surveillance in established CMMI Based - Performance 
Management database 


¢ Classification and weighting of surveillance results 


¢ Categorize surveillance to related element 
«Customer Outcome (CO) 
¢Memorandum of Agreement Task (MOA) 
“CMMI PA/P Se ES 
- Supplier Process =o me 


«Process Risk 
¢Development/Production Phase 
«Strength or Weakness 


DCMA CBPM database establishment 


Each database record links the following: 


¢ Date and individual 

¢ Project 

¢ Supplier Development/Production Process 

¢ DCMA task/action 

¢ Customer Outcome — Memorandum of Agreement element 

¢ CMMI Process Areas/Practices 

¢ Observations 

¢ Defect with weight 

¢ Root Cause with weight 

¢ Corrective Action or 
Improvement Opportunity 


DCMA CBPM database establishment 
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DCMA Capturing Surveillance Results 
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Facility Name Initiator | PA Weighting 
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PBM Element [§§ §!s-Focus Team, IPT, DDT, Working Group participation 


Customer mandated in QALI. Perform a Process Review of all customer mandated task in the 
QALI. % of Planned tasks performed. Includes weekly S¥¥ test status meetings and Design 


Product/Service/Task Process Area 
| Test Readiness Review (TAR) | Bg CM\SP 1.1, [ CM 
| _Configuration Management | Risk Rating [| Mm 


Identify the configuration items, components, and related work products that will be placed 
under configuration management. 


DCMA Capturing Surveillance Results 
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DCMA Map & Analyze Surveillance Results - Step 4 


Defense Contract Management Agency 


¢ Results mapped to CMMI Process Area Practices 


¢ Analyze Data 


¢ Supplier Critical Process Areas evaluated 


¢ Other linked elements analyzed 
¢ Appraise Applicable Critical Process Areas 
¢ Define Strengths or Weaknesses 


_Step1 Plan & Prioritize Surveillance | Step2_ 
PA’ ae Proces 


Implement Surveillance 
FORMANCE s Reviews 


. IMS, CRITICA\ [pax 7 3 | 
¢ Program Risks reassessed 255 pa 


STRENGTHS 
EAKNESSES 


PERFORMANC! 


DCMA Report Surveillance & Make Recommendations 
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¢ Generate Report 


«Supplier processes that did/did not have an issue 


¢CMMI Process Areas & Practices that did/did not had 
an issue (strengths and weakness) 


-Customer Outcomes observations performed | aes 
‘Memorandum of Agreement observations performed by 
¢DCMA Tasks performed : Customer 


«Observed issues and opportunities 
«Root Cause of issues 


¢ Recommend adjustments to Customer Outcomes 


¢ Adjustments made to Surveillance 
«Helper Guide CMMI PA/P to Task 


DCMA CMMI Based - Surveillance Framework 
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- Step1 Plan & Prioritize Surveillance 
CRITICAL 
PAST PERFORMANCE SSE EES AREA 


IMS, CRITICAL PATH 


Step 2 Implement Surveillance 


Process Reviews 
Product 
Examinations 
Data Collection 
Corrective Actions 


Record, Classify, and = Step3__ GE GES 


Categorize Surveillance Results a 
DEFECTS 


USTOMER OUTCOMES 


DCMA REPORT 


[-In terms of COST, 
SCHEDULE, TECHNICAL 
PERFORMANCE-] 


Map & Analyze 


Report Surveillance and Surveillance 
Make Recommendations Results 


DCMA Benefits of Strategy (Used) 


Defense Contract Management Agency 


¢ CMMI Based — Performance Management (CBPM) developed as a 
structured surveillance method designed to promote program 
success via critical process area - based program insight. CBPM 
helps to: 

¢ Identify and prioritize the critical processes of the program 
¢ QObjectively evaluate those processes, relative to the CMMI 


¢ Identify and assess suppliers’ process related risks along with other 
program measures 


¢ Collect, analyze and report process-related risk information and the 
impact to the program and its’ products 


¢ Supplier’s process capability, subsequent adherence and 
performance can be continuously evaluated and monitored 
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Another innovative DCMA approach for 
utilizing 


CMMI for software surveillance 


“Practical Use of DCMA’s CMMI based Risk 
Management” 


By Gary Sigmund 
Track 5, Wednesday at 1:30 
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Question ? 


Or later 
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Agenda 


e Introduction 

e Why Are We Here? 

¢ CMMI Gap Analysis With MSI Focus 
e Addressing The Gaps 

e Integration Of SSM and QE 

e Lessons Learned and Future Steps 


Mission Assurance of Software Suppliers — 
A Partnership Between Software Supplier Managers and 
Quality Engineering 
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Raytheon 
Introduction — At a Glance 


e A global technology leader in: 
— Defense, government and commercial electronics 
— Space 
— Information technology 
— Technical services 
— Business and special mission aircraft 
e 2005 sales: $21.9 billion 
¢ 80,000 employees worldwide 


e Headquarters: Waltham, Massachusetts, USA 
— http:/Awww.raytheon.com 
— Common stock ticker symbol: RTN 


Partners in customer success 
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Raytheon 
Introduction — Business Areas 


Missile Defense 


Intelligence, Surveillance and Reconnaissance 


Precision Engagement 


Homeland Security 
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Raytheon 


Network Centric Systems 


i e 


Command & Control Systems 


Missile 
Systems 


« 


Integrated Communications 


andi 
eRe ease We 


Combat Systems 


Superior mission integration for communication and information dominance 
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Raytheon 


Raytheon’s Global Presence 
Serving Customers Around the World 


e RaytheonNnternational, Inc. business development offices 

® Raytheon International, Inc. business development headquarters 
» Raytheon Systems Limited 

» Raytheon Australia 

e Raytheon Canada 


© Thales Raytheon Systems 
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Introduction - “Process Journey” Gineoel 


ore CMMI 
RTIS Policies & perioduct Sree sine IPI SW, SE &H 
Procedures TPrGeee: & Cycle Time Multi-site 


SW Level 5 
SE Level 3 


RTIS Software 


10x Baseline 


Fault Density Validation 
Improvement 


Operating 
Instructions 


eter eth Acquisition Level 4/5 
cies sacs Level 3 & Transition Managed/ 


Level 2 Defined Optimizing 
Repeatable 


Self-assessment 
Software 


Improvement 
Team Formed 


IPI = CMM-based Internal Process Improvement 
Assessment 


RTIS = Raytheon/TI Systems 


CMMI = CMM Integrated 


A long history of process excellence 
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Level 5 
CMMI 
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Why Are We Here? - 
The Burning Platform 


e Our business is evolving from a system developer to a 
Mission Systems Integrator (MSI) 


e > 70% of what we deliver comes from suppliers 

e The role of supplier management must evolve 

¢ In this presentation, = =e * 
Software Supplier > eC ———- 
Managers (SSMs) and . 
Quality Engineers (QEs) 
are being highlighted to 
address this evolution 


—. 
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CMMI Behavior/Skills Analysis With MSI Raytheon 
Focus 


¢ CMMI-SE/SW/IPPD/SS V1.1 analysis of model used to identify behavior/skill gaps 


Key Behavior/Skills Noted in CMMI Required for MSI Success 


associated with handling suppliers 
[Acquisition planning and preparation 


Receiving, storing, using, and maintaining Supplier Agreement Management Moderate Major 
acquired products 


Legend: Major = Key Role 
Moderate = Need knowledge 
Minor = Support Role 


MSI Success depends on trained skills & knowledge 
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Raytheon 
CMMI Gap Analysis With MSI Focus 


¢ CMMI-SE/SW/IPPD/SS V1.1 analysis of model used to identify behavior/skill gaps 


Key Behavior/Skills Noted in CMMI Required for MSI Success 


SSM role QE role 
Application Domain Process and Product Quality Moderate Major 
Assurance 
Customer Relations Process and Product Quality Major Major 
Assurance 
Process descriptions, standards, Process and Product Quality Major Major 
procedures, and methods for the project Assurance 


Quality assurance objectives, process Process and Product Quality Moderate Major 
descriptions, standards, procedures, Assurance 
methods, and tools 


Identifying potential sources for candidate Integrated Supplier Management N/A N/A 
products to be acquired 

Acquisition feasibility and product life-cycle | Integrated Supplier Management Support N/A 
costs analysis 


Evaluating supplier work products Integrated Supplier Management 
Monitoring supplier processes Integrated Supplier Management 
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Raytheon 
Gap Closure Process 


e Develop SSM value stream and identify performance 
attributes 


¢ Identify available training courses to supplement 
performance gaps 


e Develop program specific training model to address needed 
Skills 


¢ On-the-job training with senior practitioners 
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Value Stream of SW Supplier Excellence Raytheon 
and Associated Performance Attributes 


Talie)aaatsudceyam miley’ 


= Define ) 
= spec Evaluate Negotiate Manage Deliver 


= & REP ym 1 ke) oXey-y-\i-wam Oxe)alig- (ers suppliers Products ee 
uh 
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Raytheon 


Example of Value Stream Step One Performance Attributes 


Performance Attributes Standards of Excellence Source of skills 


Requirements (PCD) Comprehend system 
requirements and integrate 
CONOPS strategy to capture 
applicable software performance 


requirements. 


Define Spec 
& REP 


SoS Architecture Understand the key 
customer/program quality factors 


to guide tradeoffs. 


Process Capability Understand parent process 


requirements. 


Comprehend the contents of 
statement of work 


Contract Comprehend the contents of the 
contract & establish evaluation 


criteria. 


Domain experience 

OJT 

Mission Use Cases (SME) 
SE Class - Reqts Allocation 


ATAM, JTA-A 
SEI Certified Architect training 
SE Class on Architecture 


OJT, OCD, FCS SDP 
DOORS Training 


Specific standards of excellence associated with each 


performance attributes 
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Example of Program Training Model 


IMS 
(Open 
Plan) 


Source Selection 


SSAC 
Training 


SSEt 
Training 


O NR Rc —~ | 
WA {| OT sR OT ORT UT CRT | es 
Roof of rc f R of R | Rf Re | | a 
wap e TT ae el 
wap [se 4 | aewees! 
wa fe Rk PR ee 
N/A NR R 
ee ee es 
Rf rc { reo { Rf R | Re | rR | | ee 
WA {| OT sR OT RT OT CRT RT Pe I i ee 
ee ee ee es es 
a es a CE (a EE ee eee eee 
Rf rc {of R_ f R | Rf Re | ee es es 
WA {| OT NR OT RTT CRT ee es ee 
WAS |) Ne ke ee ee es es 
WA [| rc [| nek fo oR | R | | ee 
NWA fT CT NR OP OR OR re | RE | RT OR RT 
NWA {| TR | RT RR 
a a a Ee ae ee |e eee eee eee eee 
Se EE RR a (EE 
NWA ff OP RP OR RT 
a a A (<a  (  e 
NA} a EE ___id_ 
a a SR CS (A; SE SE (RE (ee (ee 
a a (ES eS ee ee eee 


Training courses to supplement performance gaps 


Raytheon 


Training Classes 


Specific training classes for 


those engineering 


disciplines to perform the 


tasks at hand. 
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Addressing The Gaps by On-The-Job- Raytheon 
Training 


e Mentor under a senior practitioner 
e Shadow a senior SSM on different meetings and activities 


e Familiarize the processes and procedures through self- 
learning 


e Begin with lesser responsibility on a smaller supplier 
e Increase responsibility with demonstrated performance 


Mentoring and Hands-on experience to address gaps 
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Raytheon 
Integration Of SSM and QE 


e Advantages of a Integrated Team 
— Different perspectives of supplier activities 
— Review of critical metrics 
— Participation at reviews 
— Review of documents to ensure requirements compliance 


e Performance Multiplier of an Integrated Team 
— SSM focus on cost, schedule, and technical performance 


— QE focus on process compliance through audits and surveillance 
checks 


Integrated team provides the best combination of 
supplier management 
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Raytheon 


Monitoring of Supplier activities 


Compliant 

Open findings with acceptable plan 

PNY Red [Unacceptable resolution plan or 30 days past due Data as of: 
1/28/2006 

po RY Blue [Rescheduled audit 

Not audited U 

Process audit still in work 


Supplier/] FCS Previous 
Process Audit Area LSISDP | SDP Year Jan | Feb Apr | May | Jun | Jul Aug Sep Oct Nov | Dec | Latest Status 


2005 
-Recording rationale 


-Access for Acquirer Review : 
-Software Cost & Schedule Estimation 


-Software Measurements 
-Group Coordination & Communication 


A] SA 
Hm] CO] o 
npaAp sR 
Hm] cl] o 
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ke 
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Bl] wth 


[Defec Managemen i 
[Software Development avonment fs 
[Software Process Management —__[f 


Example of an audit plan t 
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EEE EERE 
FlEEIEEEEEEEHEE 
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op) 
w 
NT OD 
w 


eee ee 
ensure process compliance 


Oo 


Audit plan to ensure comprehensive coverage of process 
compliance 
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Raytheon 
Software Metrics Analysis Process 


Validation 


Review & Comm. 


North Texas Metrics Process 


Metrics Analysis to monitor and track performance 
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RaySMART - Checklist 


ey VRevilew Gritenia 


EN AMPLE 


General 
Comments: 


a 
P= 
o 
oO 
wn 


Review 


6.0 Views & Patterns 


4.0 System 5.0 Software 6.0 Views T.O0 Life-Cycle %.0 Feasibility 
Requiremeats Architectere Patteras Plas Rationale 


Use Case View TECHNICAL 
a; 6.1.1 Are the use cases in the use case view consistent with the logical view sequence diagrams? 
= 4 6.1.2 Does the use case view cover key behaviors? 
| 6.13 ls this view complete with respect to the suppliers completeness criteria as documented in the supplier tailoring of the GSS SADD 


62 Logical View 


a | 6.2.1 


€ ~ 


Er 

a «625 
6.2.6 

a 

: 5 £27 
6.2.8 


Appendix C? (Use Case Diagrams and Use Case Text) 


Are the supplier use cases consistent with the GSI use cases as documentin the GSS Operational Concept Document? 


TECHNICAL 
Are the top-level subsystems and classes defined? 


Are the relationships among the top-level subsystems and classes described? 


Is the control architecture captured (in terms of state machines for individual classes/objects, and via an Object Communication Model or 
the equivalent)? 


Are the internal interfaces, as captured in the interaction diagrams, described? e.g., this message contains Time of Day. 
Are the external interfaces on the Supplier Sequence Diagrams consistent with the CEEU/CREU IRS? 

Are the logical view diagrams internally consistent with one another? 

Are safety and mission critical classes clearly marked? 


Are the Supplier Sequence Diagrams (e.g., those presented atthe Software Architecture Evaluation, and more) consistent with the GSS 
System Model Segment threads? 


Is the mapping between the GSS SADD Logical view elements (Aggregate Software Elements) and the Supplier Logical view elements 
clear (subsystems, classes)? 


[=| 


ool to consistently capture reviewers’ feedback 


Raytheon 
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Audits and Document Review 
Fiome| PREG (ERAQRDESECORUSDRL i So — 


Overview| Assignments Updates Checked-Out Work Workspaces Meetings Notebook papers | 


Welcome Jack S Lee Recently Accessed: ~| Last Login 
Search within my mar 
ir Assignments Current View: Open 
‘Name Actions _| Subject State ‘Deadline | Done Assigned 
@ SDRL Delivery Notification SW See Actions S034i Completed Oct 2, 2006 Sep 22, 2006 
@ SME Review of SDRL See Actions S077 Under Review Oct 5, 2006 Sep 15, 2006 
@ SDRL Delivery Notification SW See Actions S034i Completed Sep 25, 2006 Sep 15, 2006 
| View All | 
Completed_Reviews: 
\y Updates Current View: Most Recent + & 


=5| 
Copy Add to Hot Links 


Name Number/ CAGE Code | | Actions State Last Modified Version| Contex 


No Items to Display 
View All 


SME_Comments: 


List of document reviewef | 5 psprove 
© Reject 


Open_Reviews: 


Raytheon 


Document Review Tool 


Completed Role Reviewer Vote Comments 


Reviewer 
Jack S Lee 
Doug Buchanan 
Daniel E Campo 


Open Task 

SME Review of SDRL 
SQE Review of SDRL 
SQE Review of SDRL 


© Conditionally_Approve_Resubmit_This_Submittal 
© Conditionally_Approve_Next_Submittal 


Electronic collaboration system to ensure timely 


completion of document reviews with the right reviewers 
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Lessons Learned 


e There is a massive amount of information 
and data required to manage multiple 
Suppliers and appropriate tools are critical 


e Performing supplier management in an 
MSI role is not the same as simply scaling up 
the support roles of a “traditional” developer 
program 


e Importance and cost of communication and 
coordination 
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Lessons Learned - Continued 


¢ Lessons learned are captured by stages to ~ 
enable continuous improvement 


e Review Tools are mandatory to save time 
and be consistent 


e Working with a diverse team with different 
culture and in different geographic locations is a challenge 
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Raytheon 
Summary 


e The roles of SSM and QE have evolved over time to better assess and 
address supplier management 


e Evolution not as a “scale up” from a development program, but to 
include new activities and interactions 


e In addition to SSM and QE, other disciplines contribute to supplier 
management such as supply chain, contracts, etc. to make up the full 
supplier management team 


e Feedback SSM training needs to organization for gap closure 


e Re-evaluate against the CMMI model to ensure that all elements are 
appropriately addressed 
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Raytheon 
Contact Information 


¢ Jill Brooks 
— 972-344-3022 
— Jill A _Brooks@raytheon.com 


e Jack Lee 
— 972-344-6567 
— Jack Lee@raytheon.com 
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Questions 
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vac Systems Engineering 
Revitalization using CMMI® 


Michael T. Kutch, Jr. Sandee Guidry 
Director Engineering Operations, Code 09K SEI Authorized CMMI Lead Appraiser 
Chief Engineer Code 70E SEI Authorized CMMI Trainer 
Intelligence & Information Warfare Systems TECHSOFT, Inc. 
Department 


Engineering Process Office 
SPAWAR Systems Center Charleston (SSC-C) SPAWAR Systems Center Charleston (SSC-C) 


NDIA CMMI Technology Conference, November 15, 2006 


N65236-ENGOPS-BRIEF-0025-1.0 ‘Wecusort 


Technical Software Services, Inc. 


VE 


Systems Center 
Charleston 


Presentation Outline 
> Introduction 
>Revitalization Effort using CMMI® 
>Training 
> Summary 


WecHsort 
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VE. Introduction to SSC-Charleston 


>Where we fit 
>What we do 


>Who we are 


WecHsort 
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VE Where We Fit 


Systems Center President 
SPAWAR Secretary of Defense 
Space and Naval Warfare 
ia ia Secretary of the Navy gummi Other DoD 


(o9 N[0) ASN (RDA) 
Fleet Support PNexelUiiitel a 


NETWARCON] MARCOR ) SPAWAR NAVSEA NAVAIR NAVSUP NAVFAC 


ADDU for C4I San Diego, CA Washington, DC Patuxent River, MD Washington, DC Washington, DC 


NAVSEA NAVAIR 


SYSCEN SYSCEN SYSCEN SFA 


San Diego, CA New Orleans, LA Norfolk, VA Chantilly, VA 
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SYSCENY, 
Charleston, SC 


Network Centric 
Enterprise 


Swe 
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Technical Software Services, Inc. 


SPAWAR 
lA What We Do 
Systems Conter Oxo] aTar=reataremantemuycclaitelalccis 


Mission- We enable knowledge superiority 
to Naval and Joint Warfighters through the a? a eet 
development, acquisition, and life-cycle \ te: Technology 
support of effective, integrated C4ISR ; 
Information 
Technology, 
and Space 
capabilities. 


Body Worn 
Variant 


ere 


NETCOP-Network Common 
Operating Picture 


= 4IR Pocketscope | 


Vision- 
Fully Netted 
in Three 


“fe Rapid 
i e' Prototyping 


Speed to 
Capabilit 


We are the 
Principal C4 
Acquisition 
Engineering & 
Integration 
Center on the 
East Coast 
& Principal ‘ 
C4ISR ISEA for fax 


the Navy 


WecHsort 
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VE Who We Are 


Systems Center 


Charleston A Large Systems & Software Engineering Organization 


Computer 
Science/ 


=tatetiatcr-dalale p 


Over 70% of 
workforce is in an 


Science & 
Engineering, 


1049 


engineering or 
computer-related 
discipline 


wyye—Contracts & Supply, 112 


©————__ Other, 174 


Program Mgmt, 106 Logistics, 79 


¢ The solutions to the global war on terror developed by SPAWAR 
result from good systems and software engineering. 


¢e Systems engineering is our core competency. 
¢ Total workforce of ~ 2,300 employees. 


Wecusort 
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VE. SE Revitalization Effort using CMMI® 


Systems Center 
Charleston 


>Vision 

> Organization 
>Plan 
>Process 
>Tools 


WecHsort 
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VE Vision 


Systems Center 
Charleston 


¢ Vision 
= Develop and maintain a World Class Systems Engineering 
Organization 


¢e Approach 
#» Achieve Command-wide operational consistency 
=" Based on ISO/IEC 15288 — systems engineering 
=" Based on ISO/IEC 12207 — software engineering 
= Based on implementing CMMI® “Staged Respresentation” 


= Measure using best practices of CMMI® “Continuous 
Representation” 


¢ SSC-C commitment reaffirmed and formalized in 
Process Improvement Policy, 11 December 2003 


WecHsort 
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SPAWAR 
Vv, 


Organization for Implementation 


Systems Center 
Charleston 


Team Chairman 


= sEIPartner 
“‘Wecusort 
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SSC-C SE Revitalization Plan 


x 


Systems Center 
Charleston 


a (sYo0 l=] a1 ecme) mete) Or Ome) om acc\Viitclipsclilela 


Training / Education 
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SSC-C SE Instruction Intro to P| WBT CMMI® Level 2 
SSC-C SE 
Seine Sey SE Fundamentals 
SSC-C SW-Maint SE for Managers 
Process Manual 
Project & Process 
EPO Website Workshop 
ePlan Builder Intro to Software Engr. 


Architecture Dev. WBT IT Tools 


v 
| Underway 
[EE Completed/ongoing Certification/Degrees aoa 
. “Wecust 


SOFT 
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CMMI® Level 3 


Balanced Scorecard 


Lean Six Sigma 


Integrated Product 
eams 


— 
< 


‘E 


Systems Center 
Charleston 


SSC-C Engineering Process Guidance 


- 5 
To be replaced with 
SSC-C SE/SW Instruction 


Policies mandate the use of ISO/IEC life cycle processes 
and CMMI best practices 


ISO/IEC 15288 ISOMIEC 12207 


System Life Cycle Software Life Cycle 
Processes Processes 


CMMI® for SE/SW 


5 a g 
SSC-C Systems SSC-C Software T 
; op Level 
Engineering Process Engineering Process 
CMMI® Supporting Process Areas to SE/SW Processes 


m2 Sse CMMI° to 
| OPF | OPD | OT | Measure & Assess 


Processes 


eM 


Wecusort 
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Classic System Engineering “Vee” Diagram 
Aligning SE with CMMI and Process Improvement 


Systems Center 
Charleston 


CMMI Process Areas 
etter align with SE “Vee” Demonstrate and Validate 
qu Pp System to User Validation 


Validation Plan 


Perform System Verification 
to Performance 
Specifications 


Develop System 
Performance Specification 
And System Validation Plan 


Expand Performance 
Specifications into Cl 
“Design-to” Specifications 
and Cl Verification Plan 


Assemble Cls and perform 
Cl Verification to Cl 
“Design-to” Specifications 


Evolve “Design-to” 
Specifications into “Build-to” Inspect to “Build-to” 
Documentation and Documentation 

Inspection Plan 


Fabricate, Assemble, and Code to “Build-to” 
Documentation 


time 


Design Engineering 


———_—_-> 
PP PMC CM REQM -PPQA MA SAMs RSKM_—DAR © 


Wecusort | 
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Systems Center 
Charleston 


Engineering Process Office Website 


| Address [2 https: }corpweb2, spawar.navy.mil/cmmi/standardprocesses/standardprocesses. aspx | Go | 


SPAWAR 


Systems Center 
Charleston 


Navigation 


Calendar 


SSC-C Standard 
Processes 


Systems 
Engineering 
Process 
Software 
Engimeering 
Process 

life Cycles 


Tailoring Criteria 


Process Areas 
Projects 
Process 
Improvement 
Teams 
Organizational 
Measurement 
Repository 
Training 


Innovation 
Pragram 


SSC-Charleston Engineering Process Office 


EPO Home | ePlan Builder | WBT Courses | eWBS | Contact EPO | CorpWeb 


SSC-C Standard Processes 


Currently, the SSC-C Standard Processes contain policies, process manuals for CMMI®-SE/SW Level 2 
and Level 3 process areas, and select SOPs. The standard processes for Systems Engineering and 
Software Engineering provide SPAWAR - E 

detailed procedures for ~ y, SSC-C Engineering Process Guidance 
accomplishing tasks within ee 
these respective disciplines. "=<" PAWAR Instructic eer SPAWAR Policy 

The 3 top-level standard = oo: 


engineering processes are: Tobe replaced wah 
sscco SESW instruction 


Policies mandate the use of ISO/IEC life cycle processes 


e Systems Engineering and CMMI best practices 
e Software Develooment ——_—_—_ 
e Software Maintenance | | . 


These processes were 
derived from the ISO/IEC 
industry standards to address 
the typical engineering work 
performed by ssc-c, 
Additional SSC-C standard 


SS8c-C Software 
Engineering Process 


Top Level 
SSC-C Processes 


a2 
S8C-C Systems 
Engineering Process 


= 


& 


processes have been ; RD | 


developed to further refine 
these top level engineering 


processes and to support the process areas of CMMI®, The graphic depicts the derivation of the SSC-C 
standard processes, 
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Related Links 


Process 


Improvement 
Communication 
Plan 


SSC-C Artifact 
Submittal Form 


a 
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] => sEIPariner 
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VE ePlan Builder Tool 


vy, ede  ePlan Builder © 
id Systems Center 


Electronic CMMI® Compliant Documentation Application 


Charleston 


~ Sponsored ed by th the Di rector of E Engineering Operations (O9K) - Mi chael Kutch 


ePlan Builder tool 


= An interactive, web-based application that leads the user through 
a structured interview process (like TurboTax®) to generate a 
CMMI®-compliant plan 


= Includes standard, consistent text 


= Generates an initial project-specific document 
e Project Management Plan (with Work Breakdown Structure) 
¢ Configuration Management Plan 
¢ Process and Product Quality Assurance Plan 
e Requirements Management Plan 
¢ Measurement and Analysis Plan 
¢ Systems Engineering Plan (DoD SEP Format) 


SEIF 


Wecusort 
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VE. Training 


Systems Center 
Charleston 


> SE Training Architecture 
>Process Improvement and CMMI® 


> Systems/Software Engineering Classroom 


>Web Based Training (WBTs) 


WecHsort 
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What do we need to be world class? 


¢ All employees need a basic understanding of 


process improvement 


All project teams need to fully understand the 
CMMI® model (all sroces-es-alklevcis) 

=" To understand all of the best practices and maturity levels 

= To comply/prepare for DoD and NAVY policy 
All project team members and supporting personnel 


need to know how to perform the standar 
processes and best practices required 


= How to do good SE, CM, QA, Planning, Measurement, Risk, ... 


¢ To properly prepare for and complete an 


assessment or appraisal, — project team members 
need to map the project work products to the 
practices assessed. 


A atst1-m al-t-Le tomer: lam el-me(-Je](eit-emlam-Mmir-lialialem-lcevalic-reqielas 
‘WecHsort > 
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W, Training Architecture 
Foundation of 
PI and CMMI® | wer | —= 


Core SSC-C project 
and engineering 
processes 
(Level 2 and 3) 


gr cine ; Quality Engineering Requirements Analysis ) 
available on-site classes Configuration Mgmt 


} 


L] Existing 
L] Existing - revise 
L_] New - develop 


Prepare Projects 
[J New - buy 


for BSC or SCAMPI 


Wecusort 
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Supporting Tools 


Quality Engineering Requirements Analysis 
Configuration Mgmt 


L] Existing 

a] Existing - revise 
L_] New - develop 
[J New - buy 


= SEI 


Wecusort 
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W, Process Improvement Training 


Systems Center 
Charleston 


Intro to Process Improvement 


= Over 950 people trained parent 
= Provided via WBT Trained 

: A e)e-l-lar-yarer-laler= 
= Now Required for all employees over 3800 * 


CMMI® 
= SEI Intro to CMMI® 
¢ Over 300 attendees to date 
=» SSC-C Level 2 Processes overview 
Project Management/Project Monitoring & Control 
= 730 people trained 
Process-specific Workshops (CM, QA, REQ, M&A) 
= 450 people trained 


* This accounts for many employees attending more than one course 


WecHsort 
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Systems Center 
Charleston 


SPAWAR 
WA introduction to SSC-C Process Improvement 


eTraining 


O Courseware Operations 

O Course Introduction 

C1 Introduction to Process Improvement 

O Terminology ‘ainally qi di 

Gar ere Originally given as a podium 
get eee ee course, converted to Web 

ED SSee Pe sn Based Training in 2004 

O Organizational Implementation N ee tor all 

O Level 2 Process Manuals ON Te QUIe 10a 
eee ee employees 

O Course Summary 


E1Partner 


ii ECHSOFT 


Approved for release to the public - October 2006 Technical Software Services, Inc. 


VG. SEI Intro to CMMI® for SSC-C 


¢ 3-day Introduction to CMMI® course teaches 


the full CMMI® model 


= Students learn how the best practices build and relate 
across process areas 


= Learn the terminology 
e SEl-Authorized instructors are well-versed in 


our implementation to augment material with 
SSC-C specific content 


® Highlight SSC-C tools and resources 
" Actively involved in projects, teams, and infrastructure 
¢Over 300 employees trained 


= Want to build a cultural foundation within the 
engineering departments 


WecHsort 
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SPAWAR Systems Engineering 
Fundamentals Classes 


Systems Center 
Charleston 


¢ Teach the Systems Engineering process 
¢ 3-day on-site, classroom course 

= Based on SMU SE Masters course 

= Customized to incorporate SSC-C SE process 

=" Over 300 SSC-C engineers trained 


¢ 1-day SE for Managers course added 


“Thought provoking, motivating, and challenging. Learning 
basic SE caused me to brainstorm many different applications 
of organized system processes. It motivated me to want to 
begin organizing its application. It also challenged me to apply 
GOOD SE practices in order to successfully be more efficient in 


the process..” 


“It was extremely beneficial to have a professor with extensive 
knowledge of the subject matter and one who could apply it to 
the SPAWAR methods.” 


Student Feedback 


Wecusort 
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VE Engineering Management Workshop 


Systems Center 
Charleston 


¢ Multi-session workshop oriented “how to” class 
¢ What is a good process? Is my process good? 
¢ How to generate project plans 
» What makes a good PMP, CM Plan, QA Plan... 
=" How to use ePlan Builder 
= Hierarchy of plans (Based on level 2 or level 3 goals) 
¢ Configuration Mgmt 
»" Are my Configuration Items (Cl’s) and Change Control adequate? 
«e PPQA 
=" How to execute a process review and work product review 
e Measurement and Analysis 
=» Are my measures measurable? 
e Requirements Management 
= Traceability - simple to complex 
¢ Monitoring and Control using Reviews 


SEIPartner 
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VE SE 101 Web Based Training 


Systems Center 
Charleston 


eIntroduction to Systems Engineering WBT 
= 10-module web based training 


= Closely aligned to SSC-C SE Process, SE 
Fundamentals Course, and ISO/IEC 15288 


= Includes hotlinks to referenced documentation 
¢ SSC-C Process manuals, policies, standards 


= Extensive branching for more detail 


— 
7 O Originating “SEY Sree Technical 
das tt ew pterig Architecture 


een, 


SEtPartner 
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VE Topical WBTs 


Systems Center 
Charleston 


¢ Developing web-based training courses in specific 
topics 
¢ Architecture Development WBT - completed 
= Introduction to Architecture Development and DODAF 


= Designed to educate and promote value of system architecture to 
non-architects and new engineers One Architecture - Three Views 


= Tests for understanding 
¢ Risk Management 
= Risk identification a, 
# Analysis tools and techniques Pi Y | 
= Mitigation planning 
= Risk monitoring 
¢e Requirements Development 


Operational 
View 


Technical 
iew 


Wecusort 
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VE Summary 


Systems Center 
Charleston 


> Accomplishments 
>Results and Measures 
>Lessons Learned 


> Going Forward 


WecHsort 
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VE What We Have Accomplished 


Systems Center 
Charleston 


¢ Process Focus 

= Defined Policies and Processes 

# Aligned with DoD and SPAWAR guidance 

= Aligned with industry standards and CMMI® model 

= Built organization structured around processes and process improvement 
¢ Training is Critical 

= Providing Fundamentals of Engineering for new and old professionals 

=" Developed web-based training for “self-paced” and refresher training 

" Defining a structured technical career development path for engineers 
¢ Tools for the Engineers 

" Developed ePlan Builder application to generate planning documents 


= Developed templates, checklists, and web-based document repositories 
to link standards and DoD guidance to day-to-day tasks and processes 


ere la hVAr- Vale mm el-1acJi-)clalm@meNA-1iolatw-laremeoreyiantcclacm mtaleliatsr-lalare| 
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=> SEIPariner 
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VE Results and Measures 


Charleston 
¢ Selected pilot projects 
= Training and Mentoring of project teams 
e Informal Appraisals, Process Reviews, and Document Reviews 
to measure progress and identify gaps 
= Class B/C appraisals of selected projects 
" Define/review project-specific plans and procedures 
¢ Project-level Formal SCAMPIS” Appraisals (Class A) 
= Evaluated compliance with CMMI® Maturity Level 2 requirem: 
=" 8 projects appraised between June 2004 and February 2005 
¢ Command-wide ML2 appraisal in April, 2005 
¢e Immediately began ML3 effort using similar approach age 28, 2008 
¢ 4 Program/Project-level ML3 SCAMPIS™ Appraisals completed 
= Mix of Maturity Level 3 and Capability Level 3 successes 
= Preparing for Command-wide ML3 appraisal in April, 2007 


¢ Continue to internally assess additional projects against 
Maturity Level 2 and 3 best practices for Balanced Scorecard 7 
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VE Is It Working? 


Systems Center 
Charleston 


¢e Recognition of SE and CMMI effort 
= 15t Systems Center to achieve Maturity Level 2 
= 1st Systems Center to have a program achieve Maturity Level 3 (AP) 
= Multiple presenter at NDIA SE and CMMI conferences 
¢ High interest in Tools, Training, and Implementation 
¢ Business 
=» LPD 17: “They see us as a model and want to increase our efforts.” 


= Automation Program: “We had hundreds of sites and there was a need 
for a structured organization to put a ‘wrapper’ around that and control it. 
CMMI became the wrapper.” 


=» CICS: “CMMI was key to achieving the project goal.” 


» VIDS: “The VIDS failure (2000) motivated implementing CMMI because 
the team needed to change course or the customer would have no 
confidence in system development. It was a tremendous Success...” 


¢ Innovation - Code 71 is receiving USMC/ONR $1M Intelligence 
Metadata Project due to “Quantum Leap” Innovation Initiative 


=» “| don’t think we would have even been considered if we hadn’t told them 
about this Innovation Project” 


= SeIPartner 


‘Wecusort 
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VE Lessons Learned 


Charleston 

e Senior Management support is critical to success 

¢ Training 
=" Everyone needs to be engaged — “train the masses” 
= Specific training for process owners/subject matter experts 

¢ Utilize Teams (IPTS) as champions of specific processes 
= Multi-department representation 
=» Change agent mentality 
= Process-focused charters 

¢e Resource Properly 


= Implement with projects that want to improve, can benefit from efforts, 
and that recognize own weaknesses 


=" EPO staff provided skilled coaching, resources, support, and tools 
= Project members learned by doing and maintaining 
¢ Goals and Publicity 
=" Keep goals to sizable bites (projects) 
= Publicize successes; Share best practices 
Wecu SOFT . 
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VE Implementation Plan - Future 


Systems Center 
Charleston 


¢ 2007 
= Conduct interim Maturity Level 3 appraisals on projects 
= Correct findings and strengthen institutionalization 
=" Conduct Command Maturity Level 3 appraisal in April, 2007 
= Incorporate new version of CMMI model (V1.2) 
= Incorporate IPPD (Integrated Product and Process Development) 
= Push Level 2 projects to Level 3 


¢ 2008 - 2009 
= Begin Maturity Level 4/5 implementation 
" Establish/Refine organizational and project measures 


= Increase collection of project, process, and organizational 
measurement data 


= Conduct interim Maturity Level 4 appraisals on projects 


Wecusort | 
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‘E 


Systems Center 
Charleston 


¢ Aggressive SE Program 
e Industry Standards 

» Systems Engineering 

= Software Engineering 
¢ Best Practices 


Summary 


« Successes 


= April 2005 Command Achieved CMMI® 
aturity Level 2 as certified by Software 
Engineering Institute 
= June 2006 Common Information Centric 
Security Se roject achieved CMMI 
Level 3 in 16 of 18 Process areas 


= 15° SPAWAR Systems Center to 


fn TEAN_| achieve these levels 
di SX FE || * Goals 
Sees; bette | =z = World-Class SE Program 
¢ Automated Tools EPO Website ~~ | ® Support Command Balanced Scorecard 
= ePlanBuilder : naka : . ee to achieve 
= eWBS corpweb2.spawar.navy.milicmmil 


¢ Training — 1,600+ 
= SE Fundamentals - 305 


" Web-Based Training courses ° "a= 
. = SSC- CPI, intro to SE, Arch. Dev. sam 7 


kJ 
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SPAWAR 
Vv, 


Systems Center 
Charleston 


Thank you ! 


Contact Information: 


Michael T. Kutch, Jr. 
SPAWAR Systems Center Charleston 


Email: michael.kutch@navy.mil 
Phone: 843-218-5706 


N65236-ENGOPS-BRIEF-0025-1.0 


Any Questions 


Sandee Guidry 
TECHSOFT, Inc. 


Email: sdguidry@techsoft.com 
Phone: 850-469-0086 


? 
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Using Design for Six Sigma (DFSS) to 
improve the ease-of-use of a Process 
Asset Library (PAL) 


Fred Roberts 
November 15, 2006 


e Design For Six Sigma (DFSS) Roadmap 
e Six Sigma Tools —- DMAIC & DFSS 
e DFSS Phases 
4m OF) al ex~1 6) 
~” Design 
A Optimize 
~” Verify 
e Results 
e Summary 
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DFSS Roadmap 


DY =Vil ays, 


Define 
Toxo) oJ 


Define 
Objectives 


Assign 
Black Belt 


1DY-AV(-1 (0) 0) 
Charter 


Form Team 


Oe) aler=] 6) 


Ulavel=yacyeclare, 
Value 


Gather Voicel/ 
Image of the 
Customer 


Create/ 
Prioritize 
Requirements 
Create/ Select 
Design 
fe) aTex=) 4) 


Minimize Risk 


Design 


WVEVem daly 
=) cole [U(e1 me) @ 
Process 


Refine 
Design 
Oto) alex) 014 


Ulavel=\acieslare| 
WVesVarehitey al 
Minimize 

Complexity 
(Oxo) alele lors 

mUrareadceyarsl| 
Analysis 
Minimize 

Risk 


Optimize 


Conduct 
Experiments 


Analyze and 
Optimize 
redesign 


Simulate 
=X) acolaratslalersy 


Pilot Test 
Minimize 
Risk 


Update 
(Oe) aline) mM aat-lal 


Verify 


Create Roll 
out Plan 


farsi v4) 
FMEA 


Document 
(exe) ay ine) df: a] 


Implement 
[oXo)ayige)| 
WiCexed atslaleovagksy 


Verify Long 
Term 
Process 
or-Ver-ley [i avg 
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DY VAN { @ar.-a D] Soto Weye) I= 


Process Map/ Value Stream Stage Gate 
Cause and Effects Matrix Value Chain Analysis 
PFMEA Critical Parameter Map Customer Interviews 
MSA Quality Function KJ Methodology 
Capability Analysis Deployment KANO 
Descriptive statistics Process Prioritization Concept Generation and Pugh 
Graphical Techniques Selection 
*Box Plots DFMEA 


eHistograms 
Cheers Wis) al nd (0) Ky 


eTime Series 
ePareto 
Statistical Process Control Statistical Tolerancing 
Hypothesis Testing Monte Carlo Simulation 
Central Limit and Confidence Design for Assembly 
Comparison Testing Design for Manufacturing 
Correlation and Regression 
Multi-Vari Studies 
Sample Size Selection 
DOE Full & Fractional 
DOE Central Composite 
Advanced DOE 


Reliability Analysis - Testing 
Yate Mivfexe (ti lale] 


1D) PAN Oam Koyo) oy=1 
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Design For Six Sigma: Concept 


Gather Voice/Images of Customer 


e The perception that the PAL was hard to use came from 
“hearsay.” Data on the nature and scope of the 
dissatisfaction did not exist. 


e We used the Voice of the Customer (VoC) process and 


Fs @A UIs) K0) 081s) mets) (s101N OAM \Vtollap @lkOMUlale(s)asir-lale mV alslmersll [sists 
dissatisfaction with using the PAL. 
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...65% sure people 


spend 1 hour/day lost 


in the CPF. #121 


The CPF is easy to use 


Users benefit 
from easy access 
to the CPF 
Architecture 


The structure 
is easy and 
self- 
explanatory#5 


‘The CPFisa 
fe Laxeye| 
reference for 
silss 


°CPF is online! 
It’s good!#89 


*The CPF is 
well organized 
hyperlinks and 
templates are 
useful.#4 
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The usability of the 
CPF continually 
improves 


The CPF has 
improved 
dramatically in 
the last year for 
Business 
Development#3 
1 

The CPF was 
difficult to use 
a while ago but 
now itis 
organized#5 


Because it is 
easy to use 
eee) 0) (MN Me ly=) 
the current 
document by 
reLUi Tale mi meyameal-) 


alaaitcela, 
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The CPF contains all 
necessary information 


*The CPF is 
comprehensive, 
CV(=Tavadaliare mYcole mer: Ua) 


imagine is in there, if 


VColU Marc \V(em-yalelele la) 
time to find it. #22 


OT aT=sx-} (0) o-Jave) ey el layep. 
Provides everything 


I need to do my job 


in one place Easy to 


navigate to 


templates & guides 
used. Can focus on 
Co) d@lla mes} ccrs(e Moy im dal) 


process#74 


*All in one place! 
Improvement over 
LotusNotes which 
lareXem Coxe nir-lanyg 


Image KJ: Good/Bad CPF 


BBat-mexe)an)e)c-1al-1al-JiV(-M Ol nd ame (el-t-m ale) 
need improvement. 


The CPF is 
perceived as not 
needing changes 


| really don’t 
have any 
frustrations with 
the CPF#32 


| don’t know 
what | would 
Cod arclale[=m-velelels 
the CPF#7 


¢Can’t think of 
any way to 
improve the CPF 


The CPF needs 
improvement 


a Co) ate) ce retail r4-19| 
WV(=) | me) me C=yilayexe| 
by stages 
Doesn’t spell out 
what is required, 
what isn’t; Lacks 
summary of 
what to do; All 
over the place; 

BS} Co\(-95 of] ol re (ols 
the SBUs; Lacks 
Like) WET As) 


elf you look at 
the CPF there is 
a process there 
but I don’t think 
it is organized 
correctly. #100 


At the time it 
(CPF) was the 
rated almntaliaremcome(e) 
icomet-y mime lollale may (ey 
issue with the 

jel =xe) 0) (Ma Te) are] 
forward needs 
improvement#110 


Survey of a random sample (7%) of the 
Ya) ii com ole) eluir-lilele 


e Results of Ease Of Use Survey 


Extremely poor 
Slightly poor 
So-sSo 

Quite good 


Extremely good 


20 


Usability 


AO 


Count 


60 


80 100 


Extremely Poor 
Slightly Poor 
So-So 


Quite Good 


Extremely Good 


5.0% 
8.1% 
40.0% 
41.4% 
5.5% 


How we handled the So-So group. 


The So-So or neutral group was viewed as not 
dissatisfied. This group does use the CPF and 
will have preferences on differing aspects of 
the CPF usability. They could be the “swing 
vote” if we adversely affect them. 


Design for Six Sigma: Concept 


SUT AV“ \ VM Ove) «om O@]U(-t-1 110) al) 


CoM VMcyclafcyi(srem-lc-mICOLUM VV itamaal-Mmce) Coli fare mere) in] elelal-)alcme)mexelniliilelaMadgele--t-M a e-liilc\uela ala 
elco\ViCellavemielemuuitamular-lanycolemat-t-remcome(omielel ale) ela 


Very Satisfied Neutral Not Very 
Satisfied Satisfied Dissatisfied 
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Design for Six Sigma: Concept 


100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 

0% 


CPF User Satisfaction 


@ Very Satisfied 

B Satisfied 

0 Neutral 

B Not Satisfied 

B Very Dissatisfied 


| 


Search 


Organization Content Detail 


Navigation 


| 


Change Detail Initial Training 


DY<x<J[0]an (e) mee) dare) (61 at- ban Oxo) alex) oy 


PNoWE-laler-1e ml DLO) =bat Oxo) al loliaim-Var- UNA) Ey 


Feature Relative Level (Average Utility) 
Importance 


Organization of the contents of structured around the structured around 
the Common Process sequence of how work is} | functional areas 
Framework should be ... performed 


As | go from Policy to 30.4% the detail minimal detail detail 
Procedures to Instructions | increase throughout throughout 
prefer to see ... 


O:255 Ost 0.253 


| prefer to navigate the Common | 17.9% pictures and diagrams text and links 
Process Framework using ... with links 
0.291 0.203 


et SS 7 ee md ANY, | Con 
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Design for Six Sigma: Concept 


Measure Skill transfer 
Personal Contact 


Measure Training Effectivness 


Design for Six Sigma: Design 


Refine Desiqn Concept (i.e., Can-Be 


Process Home Page Finding Process Results 


“it” exists in the CPF Optimized Search/Find 


Os are (elssom ale)m dale) 
=Uyaleiiveyaw- gets 


where to start 


Flow Diagrams 
Role Based (31) 


Asset Asset Ordering 
CAVA Vo) ge | 


HBP011 


INFENVAe Feld) are Mtatomm @xe)anlanre)al 
Process Framework 


pie knows “it” is in the [sort | 
User does not know if EE 


Help 


HBP007(5426 employees*) 


[©] DYN hom Oxo) nal pale)am cagelerstoys 


Browse Browse 
ml esl anew ive) §:4 

, Click [Click 
*Course evaluations: 
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Design For Six Sigma: Design 


UTavelsvacieclalemsclarcidlela 


e The prioritized requirements were used as a basis for 

_ formulating the questions in our survey to confirm the relative 
importance of the individual requirements. The fact that we 
were dealing with perceptions and preferences made this a 
difficult process. 

Te Malem (ole) sy-1(<reil ore msy0] 0) ele) atcrom-laleany/anlinvare (elem relaele 
Sampling(~7%) which allowed us to know who responded, 
Val (ola a=) @loxem [amcxcrelealcvalijalemigtom ele) e)0|t-li(e)ap 

e The Measurement System Analysis showed very little 
variability in responses. 
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Design For Six Sigma: Optimize 


Analyze and Optimize Redesign 

e The survey results clearly show that Initial Training was a 
source of dissatisfaction with the usability of the Process 
Framework, by nearly a factor of two. 


e Concurrent with the DFSS EOU project the Process 
@)iN/atclarexe)alijalersrem em aat-U.com [ae] @)ce)\.2)aal-val sm comih.o@) im ialcmcyb. 
categories by adding views, flow diagrams, and tweaking 
Walomcxct-laea miele late 


K Meal- secre cmesleiomie-lialialemjia(cmecelialiaremarcle mato allelalcssy| 
dissatisfaction level for our users. 
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Design For Six Sigma: Verify 


wT arcliV4oe od li] YAU Oxe)alege) Mm eatsla 


e There were five process steps in deploying the training 
with the following RPN (Risk Prioritization Numbers) 
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Design for Six Sigma: Verify 


PFMEA 


Process Potential Failure) Potential Failure 
Stepllnput Mode Effects 


No resources | nability to assess Availability of instructors/ HR Policy on Instructors 
(available , effectiveness and analysists 
improve 


Actions 
Recommended 


EO Schedule/ Plan 
or instructors 


Actions 
Taken 


Ralph 
real Ne ea fe 


Design For Six Sigma: Verify 


(@Xeyalice)m catsla 


Analyze and 
Report eal da 


Old vs. New Training 


Initial Training 


et SS SN ee a AN, | fo 
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Improved training 


@ Very Satisfied 

@ Satisfied 

O Neutral 

@ Not Satisfied 

@ Very Dissatisfied 
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Summary 


e Process Design For Six Sigma tools: 
oles 0) oko) amiat- meal manrele(s) melas e)cenulellare meals 
ma ate)’ \’samm ce) mm taT-lahvace) mi catcm ole-leutter-1-m lam aal-monlele(-) 
oles 0) ole) amiat-me(-s-J(e]ameym-vac-rerilVicmr-lalem-aniel(-Jalt 
processes 
~Z Improved the ease-of-use of our Process 
Asset Library 


| GENERAL DYNAMICS 19 


| Advanced Information Systems 


@olaleslerm lane) anntsiarela, 


Fred Roberts 

General Dynamic Advanced Information Systems 
2721 Technology Drive 

Annapolis Junction, MD 20701 


Frederick.roberts@qd-ais.com 
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Questions? 
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How to Sell CMMI to Your CEO 


in His Own Language 


Lemis O. Altan 


cognenceinc 


Improving Software Economics 


©2006 - cognence, inc. 


A Few Questions 


¢ How many of you have tried to convince your “C” level 
executives on the merits of the CMMI Model and haven't 
made any headway? 


¢ How many of you keep thinking there is a communication 
problem, if only.... | can make myself heard? 


¢ How many of you have approached your CEO regarding 
process improvement efforts from a technical perspective? 


10/10/2006 cog RENCLinc Slide 2 
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Session Objectives 


« Review the language of engineering and engineering 
processes 


¢ Learn the language of the CEO 
« Translate the engineering language to CEO’s language 


¢ Learn how process improvement efforts can be sold in 
business context — an example 
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BValelUeve(-meym taleliatcrcaalare 


Life Cycle (waterfall, iterative, spiral, v-model) 

Requirements development 

Configuration management 

Solution development 

Integration 

Interfaces 

Testing (unit, component, system, ...ability, performance, etc.) 
Builds 

Bug fixes 

Release 
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Language of CMMI Framework 


Process institutilization 
Project instantiation 
Maturity level 
Capability level 

DAR process 

Causal analysis 
Monitor and control 
Criteria 

Relevant stakeholders 
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Language of CEOs 


10/10/2006 


Revenue 

Expense — selling, administrative, engineering, etc. 
Gross margins 

Net income 


Earnings Before Interest, Taxes Depreciation and Amortization 
(EBITDA) 


Operating profit 

Cost reduction 

Revenue growth 

Return On Investment (ROI) 
Strategy 

Productivity 

Competition 
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Typical Customer Survey/Feedback Results 


¢ Quality of our products are very low 


e They may be reconsidering purchasing another product 
from us 


e They find numerous bugs 

« They feel as if they are our testing group 
¢ Our products are always late 

¢ Our products are not meeting their needs 
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State of Engineering Organization 


¢ Long hours for engineers 

¢ Fighting the same old battles again and again 
¢ Attrition is high - Losing many talented people 
« Too many bugs 

¢ Cannot get a product released as expected 


¢ Rework, rework, rework — not enough time to do It right, but 
plenty of time to do it over! 


¢« Heroes, heroes, heroes 
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Solution 1 


We are going to adapt CMMI to create engineering processes; 
we are going to institutionalize them and mature the organization. 


We will put everything under configuration management. 


We will fix all bugs before we release a product. 


It will take a long time; but it will be worth it at the end. 
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What do you think the reaction of your CEO will be? 


¢ What is this CMMI thing? 
¢ How long did you say it will take? 
¢ How much did you say it will cost? 
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¢ We find out our SW engineering costs in real $s by conducting a 
cost analysis against the industry benchmarks, based on the 
results we work to reduce the biggest cost areas. 


¢ We find out the state of our engineering practices as 
compared to the industry best practices (CMMI Model) and 
then adapt them to fix our problems based on where we can 
impact the highest costs. 
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A business approach to process improvement based on the 
language of the CEO: 


Step 1: 
Step 2: 


Step 3: 
step 4: 


10/10/2006 
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Financial Cost Analysis of engineering quality 


Gap Analysis of engineering practices against the 
industry best practices (CMMI) 


Correlation between steps 1 and 2 


Recommended Areas of Improvement for the best 
ROI. 
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Example: Company A - Situation Analysis 


* Company A, Engineering Organization is developing a Next 
Generation System (NGS) that incorporates hardware and software 
elements. 


¢ Software defect rates in previous products were high. 


¢ To develop and deliver a high quality product, Company A has 
engaged in several process improvements, including Design for Six 
Sigma (DFSS) House of Quality (HOQ). 


¢ To meet the timing requirements, impact, and budget constraints, a 
“QuickLook” assessment was performed — to determine highest 
value, engineering practice improvement opportunities and 
setting baselines. 


¢ — Effort will focus on 3 key areas: Project management, engineering 
and support areas across 5 engineering subsystem teams over a 
1-week timeframe and include a Cost of Quality analysis. 
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Underlying Quality Premises 


e The earlier quality can be driven into a product, the less 
costly it will be to deliver and support the product. 


« Average software projects spend 50% or more of their 
budgets fixing quality problems introduced earlier in the 
software engineering life cycle. 


¢ High cost of poor quality robs organizations of the ability to 
deliver more features and functionality to the market. 


e The underlying premise of process improvement Is: 


— “The quality of a product is largely determined by 
the quality of the process that is used to develop 
and maintain it.” 


Based on TOM (Total Quality Management) principles as taught by 
Shewhart, Juran, Deming and Humphrey 
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Step 1: Financial Cost Analysis of Engineering Quality 


Cost of Quality Analysis 
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Cost of Quality Analysis Objectives 


¢ Quantify in “$’s poor quality costs 

¢ Identify major opportunities for poor quality to reduce cost 

¢ Identify opportunities for reducing customer dissatisfaction 

¢ Identify threats to product salability 

¢ Expand budgetary and cost controls 

¢ Stimulate improvement through publication 

¢ Improve ability to deliver more to the business/market 

¢ Reduce the functionality gap versus the competition 

¢ Provide executives with tangible reasons to invest in 
software engineering improvement 
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OFo}-) mo) MOVE: Ti Ava (01010) Ore) alex-) 0) 


¢ Developed by J.M. Juran and applied successfully by 
companies like Toyota Motor Corporation to achieve 
competitive advantages through the development of better 
quality products 


¢ CoQ represents all costs associated with poor quality 


Prevention Costs 


7 Costs to Achieve Quality ¢ 


Appraisal Costs 
Cc 0Q pp 


Internal Failure Costs 


Costs Due to Poor Quality 4 
External Failure Costs 


Sv GPogg!Fan’s Quality Handbook COgnencCeinc Slide 17 
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Example - Company A, Engineering Org. Cost of Quality Results 


Summary 

¢ Total Organizational 3-Month Costs: $3, 648,000 100% 
¢ Software Development Costs $1,060,000 29% 
¢ Quality Costs $2,588,000 71% 
¢ Prevention Costs $200,000 5% 
¢ Appraisal Costs $181,000 5% 
¢ Internal Rework Costs $725,000 20% 
¢ External Rework Costs $1,482,000 41% 


Yearly Organizational Cost Savings Opportunity $3,094,000 30% 


Based on Company A SW Eng. budget of $10M/year reducing CoQ to 40% 


¢ Company A, SW Development Quality of Data Low/Average 
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Example - Company A, Engineering Cost of Quality Results 


Development Costs = 29% 


Costs incurred to directly develop the product, or interim work 
products that lead to a product. Does not include any reviews, testing, 
or rework 


Examples: 
¢ Capturing/Documenting ¢ Managing the project 
Requirements * Creating user documentation 
* — Architecting the system * Creating installation guides 


¢ Designing the system 
¢ Writing and unit testing code 
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Example - Company A, Engineering Cost of Quality Results 


Prevention Costs = 5% 
Costs incurred to keep failure and appraisal costs to a minimum 


Examples: 
¢ Quality planning ¢ Requirements reviews 
¢ Software quality assurance ¢ Design reviews 
¢ Software configuration ¢ Code reviews 
management © SCM tools 
* Supplier capability assessments =. External process assessments 
* Quality training * Process improvement efforts 


¢ Software reuse 


10/10/2006 cognenceinc Slide 20 


©2006- cognence, inc. Improving Software Economics 


Example - Company A, Engineering Cost of Quality Results 


Appraisal Cost = 5% 


Costs incurred to determine the degree of conformance to quality 
requirements 


Examples: 
¢ Purchased software testing ¢ First iteration integration 
¢ Defect reporting/tracking testing 
¢ Test automation software ¢ First iteration system testing 


¢ User acceptance testing 
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Example - Company A, Engineering Cost of Quality Results 


Internal Failure costs = 20% 


Costs associated with defects that are found prior to transfer of the 
software to the customer 


Examples: 
¢ Design corrective action ¢ Defect reporting/tracking 
¢ Design re-reviews ¢ Defect fixing 
¢ Purchased software corrective ¢ 24 and subsequent integration 
action testing Iterations 
¢ Purchased software re-test ¢ 24 and subsequent system 


testing Iterations 
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Example - Company A, Engineering Cost of Quality Results 


External Failure Costs = 41% 


Costs associated with defects that are found after the software Is 
shipped to the customer 


Examples: 
¢ Next release defect rework ¢ Contract penalties 
(maintenance) ¢ Lost customers 
« “Re-engineering” ¢ Lower marketplace 
¢ Technical support personnel perception 
e Software returns ¢ Loss of pricing power 
e Lawsuits 


¢ Lostsales 
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Step 2: Gap Analysis of Engineering Practices Against the Industry 


Best Practices 


CMMI Based Gap Assessment 
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Engineering Process Improvement Premise 


Capability Maturity Model Integration (CMMI®) incorporates 
systems and software engineering industry best practices 
that are widely recognized to deliver quality systems. 


In performing the “QuickLook” Company A process 
assessment, the CMMI will be used as the reference 
model. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment 


Process Areas selected for “QuickLook”: 
¢ Engineering: 

— Requirements Development 

— Requirements Management 

— Product Integration 
Verification 
¢ Project Management: 

— Project Planning 

— Project Monitoring and Control 
¢ Support: 

— Configuration Management 

— Measurement and Analysis 
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Example - Company A, “QuickLook” Engineering Processes 
Assessment Results 


Capability 

Level Process Area 
Requirements Development (9 practices satisfied out of 12; 3 not rated) 
0 Project Planning (6 practices satisfied out of 14 ) 


0 Project Monitoring & Control ( 4 practices satisfied out of 10 ) 

[2 [Retienens eragenentapsrenss) 
pO Verification (4 practices satisfied out of 8 ) 

[2 [een nergenentsmaseanat) 
po Measurement and Analysis (0 practices satisfied out of 8 ) 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Results 


CMMI Generic Goals and Practices — 


@ S = Satisfied @ NS = Not Satisfied 
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Step 3: Correlation Between Cost of Quality and Gap 


Analysis of Engineering Practices Against CMMI 


Correlation of Two Assessments 
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Example - Company A, Correction of Engineering Costs and 


Process Gaps 


Requirements Effort % Benchmarks “Quick Look” Engineering Process 
Assessment Results 


Capability 
Level Process Area 
1 


Requirements Development 


(9 out of 12 practices satisfied; 3 not 
rated) 


Requirements Management 
(3 out of 5 practices satisfied) 


Average Leading Company A 
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Example - Company A, Correction of Engineering Costs and 


Process Gaps 


Design Effort % Benchmarks “Quick Look” Engineering Process 
Assessment 


Capability 
Level Process Area 
1 Product Integration 
(9 out of 9 practices satisfied) 


Average Leading Company A 
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Example - Company A, Correction of Engineering Costs and 


Process Gaps 


Testing Effort % Benchmarks “Quick Look” Engineering Process 
Assessment Results 


5 
30) _ 
or 
5 Level | Process Area 
15} az Verification 
10: (4 out of 8 practices satisfied) 

5, 
0 
Average Leading Company A 
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Example - Company A, Correction of Engineering Costs and 


Process Gaps 


Project Management Effort % Benchmarks “Quick Look” Engineering Process 
Assessment 


Capability 
Level Process Area 


Project Planning 
(6 out of 14 practices satisfied) 


Average Leading Company A 


Project Monitoring & 
Control (4 out of 10 practices 


Satisfied) 
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Example - Company A, Correction of Engineering Costs and 


Process Gaps 


SQA % Benchmarks “Quick Look” Engineering Process 
Assessment 


Capability 
Level Process Area 


Configuration 
Management 
(5 practices out of 7) 


Average Leading Company A 
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Step 4: Recommended Areas of Improvement for the Highest ROI 


Risks and Recommendations 
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Example - Company A, Recommended Areas of Improvement 


Koyama atom ml Celatcxoi@ nce) 


Major Areas of Risk - 1: 


¢ Most widely-recognized software experts view project 
management as a critical capability in delivering quality 
software. Project planning and project monitoring and control 
is weak in Company A SW organization. (CoQ PM cost 
allocation confirms this observation.) 


¢ Effective measurement and analysis (M&A) is almost non- 
existent. Without M&A it Is very difficult to gain insight into the 
software development processes, products, and projects, and 
take corrective actions when necessary. 
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Example - Company A, Recommended Areas of Improvement for 


the Highest ROI 


Major Areas of Risk - 2: 


¢ Verification activities at the component and sub-assembly 
level need to be better planned and implemented. Unit and 
integration tests must be tied back to the requirements 
allocated to the software components. 
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Example - Company A, Recommended Areas of Improvement for 


the Highest ROI 


Overall Recommendations - 1: 


¢ Take steps to apply a level of discipline and rigor to software 
engineering project management in-line with the DFSS and 
systems engineering improvements already installed 


— Thoroughly plan the software activities and deliverables 
as related to the system requirements allocated to 
software 


— Properly schedule the software projects — do not allow 
“un-tethered” activities to remain off schedules 
— Plan and implement a measurement and analysis 


capability that addresses process, product, and project 
dimensions 
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Example - Company A, Recommended Areas of Improvement for 


the Highest ROI 


Overall Recommendations - 2: 


¢ Institute more formal software work product inspections and 
track results. 


¢ Continue to apply the successful DFSS and systems 
engineering techniques to drive systems requirements to the 
software components where verification can take place 
against those requirements. 
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Questions and/or Comments? 
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Supporting Details 
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Characteristics of Average/Leading Benchmarks 


Average (CMM/CMMI ML1) 


Total CoQ of 55% - 70% 
External Failure Costs > 50% 
Spend 31% on Testing 
Spend 8% on SQA 


Leading (CMM/CMMI ML 3) 


Total CoQ of 40% - 50%! 
External Failure Costs 15%-25%* 
Includes process improvement, 


Spend 15% on Testing? sis so.ame 
Spend 19% on SQA? 


20% Decrease in Cost & Schedule? 
80% Decrease in Released Defects? 
65% Increase in Productivity? 


1: Dion, R., “Process Improvement and the Corporate Balance Sheet,” /JEEE Software, July 1993 
2: Krasner, H., “Self-Assessment Experiences at Lockheed,” SEI/AIAA SPI Workshop, Chantilly, VA 1990 
3: Jones, C., “Measuring Software Process Improvement,” Software Process Improvement, IEEE, 2001 


10/10/2006 cognenceinc Slide 42 


©2006- cognence, inc. Improving Software Economics 


Soo 0 


N 


= 


Company A 


Leading 


Slide 43 


Cog nenCeinc 


10/10/2006 
©2006- cognence, inc. 


Company A - External Failure % Compared to Benchmarks 
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Company A - Requirements Effort % Benchmarks 
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Company A SQA % Compared to Benchmarks 


Average Leading Company A 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Requirements Development Strengths: 


¢ NGS System Engineering process appears to be performing 
extremely well for the identification and gathering of NGS 
requirements. 

¢ [twas consistently conveyed that the to-date NGS product 
development process Is resulting in a much better defined 
product than previous efforts. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Requirements Development Weaknesses: 

« Descriptions of the subsystem-to-subsystem interface 
identified in the requirements have not yet been detailed. 

¢ The subsystem teams do not have a documented 
operational scenario (free of design), or a documented 
operational concept (design dependent). 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Requirements Management Strengths: 


e The understanding of requirements at the current phase of 
NGS is much better than the understanding of requirements 
on prior product requirements phases. 

« The system engineers and subsystem teams commit to their 
subsystem requirements through the frequent subsystem team 
meetings. 

« Changes to requirements have been documented in the 
different HOQ levels. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Requirements Management Weaknesses: 


e The requirements are documented in numerous 
spreadsheets and diagrams. Bi-directional traceability was 
not discovered. 

« Subsystem teams have not yet driven the requirements into 
the project plans and schedules, nor are they allocated to 
the work product level. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Planning Strengths: 

¢ Each subsystem team has a work breakdown structure 
(WBS) embedded in their current schedule. 

¢ The subsystem teams work within the high level product 
development life cycle described by the NGS Systems 
Engineering group. 

« The subsystem teams identify and analyze technical risks. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Planning Weaknesses - 1: 


10/10/2006 


Subsystem teams do not have a process for estimating the 
effort and cost required for the work products. 


The current processes used by the subsystem teams do not 
address the budget aspect of the project. 


Schedules exist but do not appear to address all desired 
information. 


Subsystem teams do not identify and analyze all project, 
budget, resource, and schedule risks. 


Subsystem teams do not plan for the management of data 
used or data created during the project activities. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Planning Weaknesses - 2: 

* Subsystem teams do not explicitly plan for resources (other 
than personnel) such as facilities, tools, and equipment 
reSOUIces. 

* Subsystem teams rely heavily on subject matter experts to 
perform project activities. Few subsystem teams plan for the 
future knowledge and skill needs. 

¢ Stakeholders are involved in the process, however the 
schedules provided for review do not appear to provide 
adequate visibility into the involvement of the relevant 
Stakeholders. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Planning Weaknesses - 3: 


¢ Subsystem teams have not adopted or created individual 
project development plans (at the subsystem team level) and 
processes on how they will operate. 


¢ Subsystem team personnel does not review other subsystem 
development plans, or development plans of others that 
impact the commitments to the subsystem teams’ NGS 
activities. 

¢ Subsystem teams do not all reconcile their requirements, 
activities, work products, development plans and commitments 
with available resources. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Planning Weaknesses - 4: 


¢ Commitments and other important decisions agreed upon 
between subsystem and lower level teams are not 
documented in a consistent fashion. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Monitoring and Control Strengths: 
¢ All subsystem teams track technical issues. 


¢ The subsystem teams identify and monitor their technical 
risks. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Monitoring and Control Weaknesses - 1: 


Subsystem teams do not collect and monitor project data 
against the plan. 


Subsystem teams monitor commitments made or accepted 
with outside individuals and groups, however the monitoring 
activity is not performed in a planned manner. 

Subsystem teams do not always address risks other than 
technical risks such as schedule, budget, resource, and 
other project type risks. 

Subsystem teams do not monitor the subsystem team data 
management activities. 

Subsystem teams do not monitor stakeholder involvement 
in a planned manner. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Project Monitoring and Control Weaknesses - 2: 


¢ Project monitoring and control does not appear to be as 
detailed as it should be. 


* The subsystem teams track technical issues (typically via 
the DFMEA process). However, other project issues are not 
typically recorded. 

* The subsystem teams appear to have limited capability to 
take corrective actions on issues as they are constrained by 
resource and schedule. 

* Few subsystem teams track and manage action items for 
specific issues to closure. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Product Integration Strengths 
¢ Anestablished process exists that: 
— Validates sub-system components independently 


— Requires CCB review prior to integration and delivery to 
DVT/EVT 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Product Integration Weaknesses: 


¢ Past projects have performed differing levels of product 
integration. 


¢ Some subsystem teams have procedures to perform build 
integration. These procedures do not appear to address 
integration sequencing activities or the integration of the 
hardware to the software at a planned time. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Verification Strengths: 


¢ All subsystem teams plan for informal peer reviews of 
software work products. 


¢ All subsystem teams perform informal peer reviews of the 
software work products. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Verification Weaknesses - 1: 


« The subsystem teams have not defined the verification 
methods that will be used for each hardware and software 
requirement allocated to each lowest level component(s). Past 
projects have tested the component / subassemblies using a 
series of regression tests that do not appear to have been 
explicitly linked to the requirements allocated to software. 


¢ Few subsystem teams have documented procedures for 
verifying the requirements allocated (NUD's, non-NUD's, 
interface and other requirements) to the components, 
subassemblies, or subsystems. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Verification Weaknesses - 2: 


* The subsystem teams do not collect or analyze data on the 
results of peer reviews of their software work products. 


* Few subsystem teams collect and analyze data on the 
verification activities performed on the software work 
products at the component, subassembly, or subsystem 
levels. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Configuration Management Strengths: 


¢ In most cases, code for existing products cannot be modified 
without a Pinnacle tracking number. 


¢ Release notes that detail the contents of builds are always 
produced. 


¢ Numerous documented procedures exist for creating code 
baselines and releases. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Configuration Management Weaknesses - 1: 


« The subsystem teams have not identified categories or 
individual configuration items that must be controlled at the 
subsystem team level. 


e The subsystem teams have multiple systems for use. In Some 
cases, detail of the use of some of these systems Is not 
documented. 


¢ Baselines of other project documents and internal work 
products are not been identified. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Configuration Management Weaknesses - 2: 


e The subsystem teams do not appear to track changes for 
non-deliverable project documentation, and other internal 
work products. 


¢ The subsystem teams do not perform configuration audits 
(PCA's/FCA's) on configuration baselines. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Measurement and Analysis Strengths: 


¢ Evidence was found that at least one subsystem team has 
identified measurements to collect, analyzes those 
measurements, and reports them to stakeholders. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Measurement and Analysis Weaknesses: 

¢ Few subsystem teams have established measurement 
objectives, specified the measures to be collected, how the 
measures will be collected, or how they will be analyzed 
and reported. 

¢ Few subsystem teams collect specified measurement data, 
analyze and interpret the data, manage and store the data, 
or report the data to relevant stakeholders. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Company A, Generic Goals/Practices Strengths: 
¢ No generic practice strengths were identified. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Findings 


Generic Goals/Practices Weaknesses - 1: 


¢ Organizational policies for planning and performing the NGS 
product activities were not discovered. 


¢ The plans and schedules reviewed do not address each of the 
activities expected in the process areas examined. 


¢ Some subsystem teams do not appear to have adequate 
resources (staff, tools, facilities, licenses, etc.) for performing 
to the process and creating their project work products. The 
average staff work week Is in excess of 45 hours per week. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Results 


Generic Goals/Practices Weaknesses - 2: 


¢ The organization has a heavy reliance on subject matter and 
domain experts to complete the work activities. Planning and 
scheduling of additional training required to increase the 
capability of the team does not appear to be typically 
considered. 


« The subsystem teams have not planned and scheduled for 
appropriate levels of configuration management on the 
subsystem team work products. 
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Example - Company A, “QuickLook” Engineering Processes 


Assessment Results 


Generic Goals/Practices Weaknesses - 3: 

e The subsystem teams have not planned and scheduled for the 
identification and involvement of the relevant stakeholders in 
the product development process. 

¢ The subsystem teams have not planned and scheduled for an 
objective evaluation of the process and work products against 
the standards and procedures that define them. 

« The subsystem teams have not planned and scheduled for 
process reviews that should occur with higher level 
management. 
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- Applied Enterprise Dynamics, Inc.(650) 292-0753 
Performance Intelligence, Actualized 


agenda 


Session theme: lessons for successful 
deployment / adoption: 
e Provide pmo practice insights 


Cite case studies using CMMI: case study, 
challenges, reflection, lessons learned 


e Discuss successful pmo premises 


Open forum for discussion 


- Applie seus ie 5 amic site c.(650) 292-0753 
TKIYARSYPAININS Dabforindiice Ink panne 


about pmo 


- Applied Enterprise Dynamics, Inc.(650) 292-0753 
Performance Intelligence, Actualized 


SOYA sy P4000) 


definition of PMO by PMI® 


“The Program Management Office is an 
organizational body or entity assigned various 
responsibilities related to the centralized or 
coordinated management of those projects 
under its domain. The responsibilities of PMO 
can range from providing project 
management support functions to actually 
being responsible for the direct management 
of a project” 


PMBOK 3'¢ Ed 
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pmo value creation 


¢ PMO (Program Management Office) by design 
builds on a series of best practices including 
PMI® standards. In NA, the majority of Fortune 
companies have already or in the process of 
establishing IT PMO office. 


e¢ Structure may vary; PMO Is either part of the 
business or the IT organization; regardless, its 
mission focuses on delivering business value 
with accountability for results 


- Applie seus ie 5 amic site c.(650) 292-0753 
TKIYARSYPAININS Dabforindiice Ink panne 


example of an actual pmo charter 


(education institution) 


“The Program Management Office will provide: 

¢ state-of-the-art project management software which will include 
automated project scheduling, project interdependency tracking, 
issue management, resource management and other tools; 

¢ assistance in managing risks and addressing quality 
considerations; 

e skills training for project managers; 

* consistent and effective project management methodologies; and 

° proactive communication of project and task status. 


The benefits are straightforward and include higher probability of 
success, more consistent and simplified management, and 
accountability for all results. This should ultimately lead to higher 
client satisfaction and a better place to work and learn.” 
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cmmi.role 
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Performance Intelligence, Actualized 


SOYA sy P4000) 


Cmmi. role 


Establishes an organization framework for 
successful project management practices 
Provides a roadmap for continuous process 
improvements across program and project 
portfolios 

Provides consistent measurements across 
the life cycles, e.g., SDLC, PLM... while 
providing a gauge for evolving the 
organization's maturity 


Iterative roadmap for adoption 
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Adoption and cmmMle 
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Moving 
On 


performance 


: managed 


initial 
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Organization adoption correlation 


User-adoption is synergistic — value creation is self 
renewing 


User-adoption / success metrics are proactively 
gauged 


Audience needs are clearly identified 


Inconsistent deployment success | 
eat ‘managed 


Poor product/service | initial | 
adoption ew eee Oe ee er 
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pmo case studies 
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case study 1: “major telco” 


(~$40 billion in revenues) 


Established a Program Management Office 

PMO SEPG formed with local legs at the unit level 
Governance/Audit team (post deployment) 

Super Users at the local level 

Retained a major consulting team 

ere a series of templates to support SDLC, targeting 
evel 3: 


— REQM, CM, PPQA (Proc and prod quality ass), MA; OPF (Org 
Proc), OT (train) 


— Project complexity-based requirements: High and Low 
— RTM: Requirements Traceability Matrix 

— Peer Reviews 

— Change Management process 

— Document management application 


— Project management software with resource allocation and 
time tracking 
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SOYA sy P4000) 


case study 1: what went well? 


Engagement model was clear with the customer— 
introduced “Application Front Gate” to the IT 
organization 


Consistency of templates: for example, requirements 
documents were no longer called different names: 
BRD, RD... 


SEPG ensured consistent application across a matrix 
IT organization—process exceptions had to go 
through a review committee 


Paving the road toward meaningful measurement at 
the PPM level became within reach 
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case study 1: challenges 


=a | : confusion ersisted between 
Given the culture, introduced a aie aaa 


very aggressive rollout meee 
timeline: LL 
. : r agoption: received as 
¢ Project complexity-based additional work 
requirements 
¢ RTM: Requirements Traceability asic a erceived as 
Matrix unnecessary work; most 
: completed the basic 
* Peer Reviews _ fequirements. 
¢ Change Management process 
¢ Master directory of process assets ell receive , Not well by the 
business 


/ document management tool 


¢ Project management application “~~ Saved $$$ on space needed for 


attachments; links didn’t 


(eVista Artemis) always work 
¢ Audits 
“Poor adoption due toslow 


performance; cumbersome— 


Resistance impacted productivity lowered productivity 
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audience reflection 
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case study 1: lessons learned 


Big bang introduction of people, process, and corresponding 
tools does not work all at once—cultural adoption rate must be 
taken in account 


CMMi champions were re-aligned elsewhere in the organization 
Process improvement is not defined by consultants—rather, it is 
defined by front line doers—engage cross team members 
across cycle (not only “smart” people) 

Tools’ capability and maturity must be mapped for productivity 
flow not just features 

Culture must be taken in account for an adoption rate and the 
subsequent roll-out success 


Closely evaluate capability to deliver (e.g. resources, training...) 
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ase study 2: “global shipping co.” 


($5 billion in revenues) 


¢ Established a pmo practice 


¢ Defined balanced scorecard (BSC) which defined 
the PPM objectives 


¢e Adopted ad-hoc CMMI 
e Instituted PMI standards 


e Enabled standard SDLC template in project 
management software 


e Instituted an end-to-end project governance 

¢ Piloted a project case study for the entire PMO team 
¢ Debated PMO approaches of “process and tools” 

e Promoted PMI certifications for consistent practices 
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case study 2: what went well? 


EPS (SEPG like) 


PMO Balanced scorecard directly linked to 
corporate initiatives “upfront” while providing 
a tracking mechanism across project 
portfolios (PPM) 

Paving the road toward meaningful 
measurement at the PPM level became within 
reach provided the SDLC template is widely 
adopted 


case study 2: challenges 


e Established a pmo practice PMI leading practices 


e Defined balanced scorecard 
(BSC) measurements 


¢ Adopted PMI standards cog aon cEly 
driven IT department—much 
e Enabled standard SDLC customizations were needed 
template in project ——— 
man agement software Many resisted the straight jacket 
e Instituted an end-to-end 
project governance “management had already made its” 


decision regardless—PMs 
¢ Piloted a project case study for ja salreagy om 
the entire PMO team 


e Debated PMO approaches of result, missing a anneal 
“orocess and tools” document such as RTM left 


Promoted PMI certifications 
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the organization vulnerable to 


audience reflection 
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case study 2: lessons learned 


¢ Deploying BSC and encouraging 
Standardization by adopting standards 
enabled greater PMO value creation; 
however, not considering critical 
components of CMMI such as OPF, 
RTM... left PMO more vulnerable to 
frequent project over-runs 


ws 
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pmo success drivers 
top 3 premises 


“Say not | have found the truth—rather, | have found a truth 


for the truth walks upon all paths” 
Gibran K Gibran 
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pmo and enterprise effectiveness 
is one 


¢ The marriage between people, process 
and technology can be happy or a troubled 
one. How well we understand and sanction 
these performance drivers, positive and/or 
negative, determines the effectiveness of 
PMO and the subsequent level of 
customer value creation 


iSsascience.as much.as. itis apart 
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< pability & maturity modeling can 


advance PMO performance if, 


1. PMs subscribes to a process of productivity 
2. PMs use intuitive tool--process and tools are 


aligned 


3. PMs competency is promoted and rewarded 
4. Vertical process and process inter-relations 


are complimentary 


5. Team members recognize that it is an 


evolution to higher maturity (don’t jump 
steps) 
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begin with understanding 
adoption drivers first 


1. Begin with user-adoption drivers as a layered 
foundation: Motivated adopters lead first 

2. Focus on value at the lowest denominator in 
order to drive adoption 

3. Reward a culture of ownership—motivate for 
execution 
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Contact info: 
Terry Jabali 
1201 Geraldine Way, suite 1 


ab out pre S enter Belmont, CA 94002; 


650.218-2588 
Terry Jabali 


* Terry has over 16 years of consulting expertise for creating high performance 
organizations encompassing performance management frameworks, enterprise 
applications, and business intelligence. He is the Chief Architect of Applied Enterprise 
Dynamics, Inc. (AED). He advises CxOs and managers on turnkey solutions 
encompassing program and project management, service delivery modeling for IT 
organizations, customer-centric business process reengineering, performance 
management and business intelligence with key emphasis on analytics, scorecards, and 
actionable dashboards leveraging best practices frameworks such as the Capability and 
Maturity Model (CMMi®). 


Terry serves as a Director on the board of Project Management Institute, Program 
Management Office (PMO) SIG. He is also a program certification panelist, impacting 
200,000 worldwide members. 


Prior roles: CRM Practice Head for North America, Clarify, Nortel Networks Principal, 
Computer Sciences Corporation (CSC) CRM Practice Head, Cedars (Peoplesoft) 
Leadership Development Director, American Society for Training and Development 
(ASTD-RMC) Founder, Fifth Generation Consultants Terry earned his B.A. degree in 
Applied Behavioral Sciences and completed graduate work in System Dynamics at MIT 
and Organization leadership at Stanford. Terry holds several industry certifications 
including CPM, Six Sigma Green Belt (ip), CPC, and others. 


For related whitepapers, please goto: Www.pmoiq.net 
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NDIA CMMI Technology Conference 


Looking Beyond Quantitative Defect Management 


Anil Midha 
BAE Systems, NES 
Wayne, NJ 


Presentation Agenda 


e Background 

e Issues involved 

e Where do we need to look and why 
¢ Recommendation — “The Three-Prong Approach“ 
¢ Defect insertion vs. detection analysis 

¢ An integrated approach to defect analysis 
¢ A robust causal system 

e Summary 

e Inclosing 
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Background 


e Organizations aspiring to be (or operating at) high CMMI 
maturity levels generally focus on defects 


e Typically they collect defects from work product 
inspections or reviews in development phases 


e For CMMI Maturity Levels 4 and 5, it is required to 
quantitatively manage and statistically analyze the 
defects to: 


— Understand the impact of common as well as special causes of 
variations 


— Perform root cause analysis of high impact defects 
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Issue IS ... 


Quantitative defect analysis most often focuses on “quantity” 
of defects, not other aspects of defect analysis, such as: 


e Defect prevention throughout the development lifecycle 

¢ How early are the defects detected after getting 
inserted? 

e What is the “chasm” between defect insertion and 
defect detection? 

e How do we reduce this “chasm” and thus cause left-shift 
In phase detection of defects? 

e What is the cost-effective approach to impact both 
quantity of defects and early detection of defects? 


NDIA 2006 101606 4 


Looking Beyond ... Where? 


Need to: 
e Have a defect management approach that complements quantitative 
analysis 


e Analyze defect insertion, detection, and correction process 

e Consider other significant defects beyond the software and systems 
defects detected through peer reviews and inspections 

e Review defects from more of an integrated engineering view rather 
than a single functional discipline view 

e Examine the effectiveness of the causal system 
and apply it to a broader range of anomalies and 
opportunities than just defects 
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Why Is it important? 


¢ Quantitative management and application of 
Statistical control techniques for defect analysis 
is “necessary” but not “sufficient” 


¢ Defects getting detected at later development 
lifecycle phases do not receive the scrutiny of 
the quantitative management 


¢ Anon-integrated approach of defect management misses out 
on some of the key opportunities of addressing defect 
prevention and early detection in the most cost-effective 
manner 


e Lacking a robust causal system leads to treating the 
symptoms rather than addressing the causes of deeper issues 
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The Answer Is ... 


NL 


A Three-Prong Approach: 


¢ Reduce the gap between phase of defect 
insertion and defect detection 


¢ Adopt an engineering integrated approach 
to defect analysis 


e Apply a robust causal system 
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Engineering Development Lifecycle 


Customer Req. Req. Spec Spec Arch. Arch. Dsgn. | _| Dsgn. re Cons. 
And/or Dev. |] V&V || Dev. || V&v || Dev. | | vav | | Dev. || v&av we 
User needs | | 


DD1-V V1 DD2 — V V2 DD3 —V V3 DD4—V V4 DD5—-—V V5 


e Engineering development lifecycle can be considered as a series of 1 ton 
Design-Development (DD) activities followed by a corresponding Verification 
and/or Validation (V & V) activities 


e Assumptions: 


— One or more functional disciplines (Software, Systems, Electrical 
Hardware, Mechanical, etc.) are working in parallel in various phases 


— Every functional discipline may or may not be performing a corresponding 
V&V activity after its DD activity 


— Parts of the development lifecycle may be repeated incrementally and/or 
iteratively 
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Engineering Development —- Defects Injection & Removal 


Injected Defects in Various DD Activities 


Requirements maa ( Design & 


Development & 


Corresponding = Corresponding 
V&V Activities 


: Defects Removed Through V & V Activities 
Escaping 


Defects 
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Conceptual Defect Insertion/Detection Model 


Defects Defects 


; Defects 

Inserted in Inserted in Inserted in 

Requirements Design Build 

Phase Phase 

Phase 
oO Total Total Total 
. Remaining Qp Remaining Ac Remaining 
Defects Defects Defects 
Requirements YR Yo Yc 
Phase 

Defects Defects Defects — 

Detected in Detected in Detected in 

Requirements Design Coding 

Phase Phase Phase 

Br Bp Be 


Ol, >= Br Total Defects in Design Phase, T=y,, + Q, andsoon... 
Bo<T 
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Defect Insertion vs. Detection (1) 


Customer 
And/or Req. Req. Spec Spec Arch. Arch. Dsgn. Dsgn. Cons Cons. 
User needs Dev. V&V Dev. V&V Dev. V&V Dev. V&V ‘ Vavl oct 


DD1-V V1 DD2 —V V2 DD3 — V V3 DD4 —V V4 DD5 —-V V5 


¢ A defect detected in k'" V & V activity might not have been inserted in 
the corresponding preceding kth DD activity 


Usually it is an earlier DD activity of the same or a different functional 
discipline in which the defect got inserted 


e In some extreme cases, the defect might have been inserted in an 
earlier DD activity of a different spiral/iteration 
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Defect Insertion vs. Detection (2) 


¢ l|deal Case: A defect inserted in the k" DD activity gets detected in the 
ki" V & V activity 

¢ Typical Case: A defect inserted in the k"" DD activity gets detected in 
the i V & V activity, where i" activity is an earlier DD activity in the 
time sequence 

e The gap, G, is the number of intervening V & V activities between the 
i" and the k"" V & V activities 


¢ One would like the gap, G,, to be zero 


i" DD activity k" V & V activity 
Where the defect where the defect 
Got inserted got detected 


\ The Gap, G;, 
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Defect Insertion vs. Detection (3) 


e Practically, it may not be possible to get this ideal state because it 
may be one of the intervening V & V activities (like a simulation) that 
might have been the only practical first V & V activity to detect the 
defect 

e Analyze which V & V activity “should have” detected the defect; let us 
assume it is jth V & V activity 


ith DD activity jt" V & V activity kth V & V activity 
Where the defect where the defect where the defect 
Got inserted “should have” got detected 
been detected 
The Gap, Gix 


¢ Its this real gap, G,, — let us call “Opportunity Gap”, that must be 
reduced 
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Defect Insertion vs. Detection Analysis (1) 


¢ For each significant defect in each V & V activity, we need 
to collect and analyze: 
a) Sequence number of the DD activity where the defect got inserted 


b) Sequence number of the V &V activity where the defect got 
detected 


Cc) Sequence number of the V &V activity where the defect should have 
been detected (by default, it should be same as for defect inserted) 


¢ Calculate Opportunity Gap (OG) = difference of sequence 
numbers between (b) and (c) 


¢ Best Case: OG being zero, |.e., no V &V activity missed 
detecting the defect 


e« Typical Case: OG being greater than zero; i.e., 1 or more 
V &V activities missed detecting the defect 
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Defect Insertion vs. Detection Analysis (2) 


¢ Goal: Reduce OG for all significant defects 


e Further analyze the data for: 
— Which V & V activities are able to detect more defects? 
— Which DD activities are more error prone in inserting defects 
— Which VV activities are more prone to missing defect detection? 


e This analysis should lead to strengthening: 
— Those V & V activities that are missing defect detection 
— Those DD activities that are prone to defect insertion 


¢ Over time — with appropriate process adjustments — OG 
should be reducing 
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Benefits of The Approach 


¢ By paying close attention to defect insertion and detection, 
the process changes will be applied where it would be 
most needed 


¢ Over time, most of the defects will tend to be detected in 
the earliest possible detection opportunity 


¢ Over time, most of the defect detection will have a “left 
shift” effect 


¢ This will lead to the most cost effective DD and V&V 
activities, also impacting product development cycle time 
and quality 
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Need for an Integrated Approach 
to Defect Analysis 


¢ Defects detected in the later development lifecycle phases 
are usually more complex — they impact most disciplines, 
are the most expensive to fix, and require broader & 
deeper scrutiny 


¢ Often superficial analysis leads to categorizing the 
detected defect in one or the other discipline, while it may 
have been best addressed in a multi-discipline approach 
for the most optimal solution 


¢ Addressing only a subset of root causes of the problem 
may lead to a partial or sub-optimal solution 
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An Integrated Approach to Defect Analysis 


¢ Record all integration and systems test defects in a cross- 
discipline engineering defect tracking system 


e Review the detected defects in a multi-discipline team to: 
— Analyze all possible causes of the problem 
— Assess the impact in all subsystems/components in all functional 
disciplines 
— Identify an optimal near-term solution while simultaneously 


analyzing if there is a better longer-term solution for later 
implementation 


¢ Implement the identified solution using a systems approach 


e Address root causes of the defect to implement 
preventative actions for the future 
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Need for a Robust Causal System 


e Inadequate root cause analysis may lead to: 
— Treating the symptoms rather than the problem 
— Addressing the wrong problem 


¢ A robust causal system would help uncover real causes of 
the problems so that actions could be taken to avoid 
similar problems in the future 


e Addressing root causes of the problems is one of the most 
effective defect prevention mechanisms 
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A Robust Causal System (1) 


Observations 


os 


Actions Preventive Corrective Mitigating 
Improvements Control Management 
Objectives 


Elements of a Causal System 


Reference: Card, David N. “Understanding Causal Systems” CrossTalk, October 2004 


NDIA 2006 101606 20 


A Robust Causal System (2) 


¢ For each significant defect: 
— Isolate and Identify symptoms and problem 


— Use Ishikawa diagram approach to identify all the root causes of 
the problem 


— Understand that it may produce multiple symptoms 


— Identify all possible causes that may have contributed to the 
problem 


e Identify appropriate preventive, corrective, and mitigating 
actions to address causes, problem, and the symptoms 
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Summary 


e The Three-Prong Approach needs to complement, not 
replace, quantitative defect management and statistical 
control 


¢ Defects needs to be detected in the earliest possible V & 
V activity 


¢ Adopt an integrated systems approach to address the 
significant problems identified in the later development 
lifecycle phases 


e Use a robust causal system to analyze significant defects 
and their root causes 
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In closing ... 


Reducing the number of defects 
IS as important as 
preventing them and detecting them 
at the earliest opportunity. 
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Thank You! 


Comments 
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SCAMPI® 
Appraisal Results 


Bruce A. Boyd 

Associate Technical Fellow 

Bi at=m=1e\-1 [ale @xe)an)ey-vahv, 

Integrated Defense Systems — St. Louis, MO 


Dear Lead Appraiser, 


Your recent appraisal findings briefing said that you collected data 
on the number of artifacts reviewed, the number of appraisal team 
hours, the number of projects, the number of participants, and the 
number of additional evidence items requested. 


Is there some kind of aggregate score that could be used to 
determine how effective the appraisal was? 


Curious 
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The Answer? 


IT DEPENDS! 


= Unfortunately, there is no “magic formula” for calculating the 
effectiveness or reliability of a given SCAMPI® appraisal. 


= There are some minimums for artifacts reviewed and appraisal team 
hours, but it’s just not true that “bigger is better’. 


= However, we've found that there are some key attributes that can 
serve as indicators of an effective appraisal... 
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= The following attributes of appraisals may be used as indicators of the 
effectiveness of appraisals... 


= Planning 

= Organizational Unit — Projects 
= Duration/Effort 

= Team Size and Composition 

= Team Activities and Training 

= Project Participation 


= Evidence Review 
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These indicators are derived from some actual lessons learned, some 
“horror stories” from other organizations, some conversations with lead 
appraisers, and some personal OpinNiONS. — ae 


They are not hard and fast rules, nor do 
they pretend to support or replace the 
requirements of any published appraisal 
method description. 


“Any resemblance to actual persons or 
events, living or dead, Is purely coincidental.” 


= They are just something to think about... 
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=" How thorough is the appraisal plan? 
* Planning should be done well in advance of the start date 
| The plan should reflect an understanding of the organization being appraised 
' The plan should not just be a “boilerplate” with names and dates inserted 


= The appraisal sponsor and 
other key members of the 
organization should have 
opportunities for significant 
input into and review of the plan 


| The lead appraiser should interact 
with the sponsor during planning, 
not just to get approval of the plan 


= Once the appraisal has begun, it 
should follow the approved plan 
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' How are the projects in the appraisal selected? e 7 


' The lead appraiser should assess 
how well the projects represent the 
organization as a whole 


Generally, the appraisal scope 
should not just include the projects 
offered by the organization 


|The lead appraiser should not offer a 
“site certification” — such certifications 
are not part of the SCAMPI® method 


= Projects should not be allowed to 
“drop out” once the appraisal has 
begun, nor should new projects 
“drop in” 


' Projects that have been previously appraised should not be granted a “free 
pass” — they must be appraised from a “clean sheet” 
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| Does the appraisal take a reasonable amount of time? 


* A Class A SCAMPI® SE/SW/IPPD/SS appraisal of 4-6 projects ought to take 
about 10 work-days to complete 


' Less than 5 days does not seem reasonable 
| Pre-onsite review of evidence is recommended 
| An appraisal should not stretch out over several weeks or months 


| Team work-days should not be 
excessively long 


' A planned Class B or C appraisal 
should not “morph” into a Class A 
appraisal 


' Projects should not “rework” evidence 
during the appraisal period 
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= How many people are on the team? 


= A full SCAMPI® Class A Level 5 appraisal should probably have 8-12 people 
to keep the workload manageable 


| What is the make-up of the team? 


= Team members from outside 
the appraised organization 
should be welcomed 


= A well-balanced team includes 
experienced practitioners 
representing the disciplines and 
functions being appraised 


= A mixture of experience levels 
and viewpoints provides for good 
consensus building 


= Domineering personalities should be avo 
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= What do the appraisal team members do? 


= All should be reviewing evidence, participating in interviews, writing requests 
for information, participating in characterization discussions, and developing 


findings 

| All should be operating somewhat autonomously, not as “gophers” for the 
lead appraiser 

= Team findings should be formed by consensus, not the dictate of the lead 
appraiser 


How was the appraisal team trained? 


All should have received required training in 
the model and the appraisal method — not 
necessarily by the lead appraiser 


The lead appraiser should verify training and 
other credentials of team members before the 
start of the appraisal 
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Who does the appraisal team talk to on the projects? 


Project managers and practitioners 
should be interviewed directly 


Process focals or other 
“surrogates” should not 
be primary sources of 
affirmations 


“Hearsay” evidence should 
be followed-up with 
face-to-face interviews 


Organizational EPG members should not need to participate in 
project-level interview sessions 


= Pre-appraisal briefings may address the conduct of the appraisal, but 
should not be “coaching” sessions used to provide the “right 
answers” 
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= How is objective evidence used in the 
appraisal? 


= Documented evidence (Direct and 
Indirect) should be the primary driver 
of practice characterization 


= Most effective (and efficient) appraisals 
are done in verification mode, not 
discovery mode 


= Practice Implementation Indicators (PIIs) 
of some form should be used to 
organize and present evidence to the 
appraisal team 
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= Intuition or “gut feel” should not be 
allowed to overrule the objective 
evidence 


= Counter-affirmations should not be 
ignored — they may be a source of latent 
truth 


= Artifacts should be used appropriately — 
for example, process documents are not 
direct evidence of performance 


" Artifacts should not be created just for 
the appraisal — “wet ink” documents 
should be treated as suspect 
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When evaluating whether an appraisal (planned or past) is effective, the 
organization sponsor and EPG should consider the following 
questions... 


How thorough ts the appraisal plan? 
How are the projects in the appraisal selected? 
Does the appraisal take a reasonable amount of time? 
How many people are on the team? 
| What is the make-up of the team? 
= What do the appraisal team members do? 
How was the appraisal team trained? 
Who does the appraisal team talk to on the projects? 
" How Is objective evidence used in the appraisal? 
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What you do with these indicators is up to you. 


Our experience has been that you ignore these indicators at your 
own risk. 


* When these indicators are considered, you can have confidence 
that the appraisal results represent an accurate assessment of 
your organization’s CMMI maturity level and/or capability profile. 


Will the appraisal will indicate how good you are at passing 
appraisals? 


Or... how good you are at managing projects, developing 
products, and improving performance? 
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LOCKHEED MARTIN A 


Lockheed Martin 
Integrated Systems & Solutions 


A Large Appraisal ? 


Dennis A. Ring 
SEI Lead Appraiser 


_ _M. Lynn Penn 
Director Quality Systems & Process Management 
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IS&S Benchmarking Maintenance 


Initial Benchmark 
¢ SCAMPI C 6-9 months after ATP 


One year after Initial Benchmark: 


SCAMPI B 
¢ All programs over $X contract value 
¢ Some programs between $X and $X 


— FFP, numerous subs, not co-located team, high risk, 
corporate or IS&S visibility 


SCAMPI C 
¢ Non-SCAMPI B programs between $X and $X 
¢ All programs (>$X) that are following the “primes” or 
“customers” processes 


— A PPS mapping is required and this is done for process 
risk determination 


REVISITS depend on “Process Risks” identified in previous 
benchmark — ALL PROGRAMS VISITED EVERY TWO YEARS 
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Appraisal Characteristics 


Method: 
¢ SCAMPI V1.1 (ARC Class A) 


Disciplines: 
¢ Systems & Software Engineering, IPPD, SS 


Models & Representations: 
¢ CMMI V1.1 Staged Representation 


Dates: 
e October 2005 
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Overview of Appraisal 
Appraisal Team Members 


Gene Miluk (Lead Appraiser) SEI 


Charlie Ryan SEI 
Peter Capell SEI 
Joanne O’Leary SEl 
M. Lynn Penn LM IS&S 
Bill Pohle LM IS&S 
Dennis Ring LM IS&S 
Dorna Witkowski LM IS&S 


Site Coordinators (LM IS&S): 
Dick Warren 
Connie Czaplicki 
Mike LaBarge 


LOCKHEED givin 
The Challenge 
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SEI CMMI APPRAISAL 
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Sampling Technique 


The scope of the SCAMPI was the entire IS&S organization. 
Three programs represent more than 20% of the work 
distributed across all of the IS&S regions. 


- AEHF 
- IEC 
- SBIRS (ground segment) 


Through sampling, 27 other programs were offered into the appraisal to 
validate that the IS&S processes had been truly institutionalized. These 
programs will have full artifact libraries and completed binders available 
for the appraisal team’s validation. Sampling from these programs will 
increase the coverage to 73% of the IS&S bottom line. 
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Sampling Technique 


The scope of the SCAMPI was the entire IS&S organization. 
Three programs represent more than 20% of the work 
distributed across all of the IS&S regions. 


- AEHF 
- IEC 
- SBIRS (ground segment) 


Through sampling, 27 other programs were offered into the appraisal to 
validate that the IS&S processes had been truly institutionalized. These 
programs will have full artifact libraries and completed binders available 
for the appraisal team’s validation. Sampling from these programs will 
increase the coverage to 73% of the IS&S bottom line. 
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Sampling Technique (continued) 


Review of focus programs allowed team to become familiar with terms, 
tools, and standard processes 


Allocated specific process areas to mini-teams 


Included additional programs to broaden the programs investigated to 
include all Lines of Business and multiple program types 


Reviewed program documentation (already provided in binders) 
¢ Used familiarity from focus programs to craft questions confirming 
implementation and institutionalization 


Conducted additional interviews with these programs 
¢ 10 additional programs 
¢ 42 additional people 


Mini-teams presented findings to full team for consolidation 


/ 
4, 
H N +" 


Mini-Team #1 
PP, PMC, IPM, RSKM 


Willalem Moxelaaness4 
Sy ANY im boy V/s YAN 


Deepwater 


Mini-Team #3 PAO0O 
REOM: Rito 
VER, VAL IDMO 


TTWCS 


Mini-Team #4 
iV =) 210) We) 1 Oy = DD(X) (DV) 
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Results of Broadening Coverage 


=" Found appropriate consistency of practices, 
tools, and processes across the additional 
programs 


= No major weaknesses, issues, or concerns 
were found 


=» Increased team confidence In 
institutionalization across IS&S 


LOCKHEED MARTIN A, 
Carmccice Viellon 
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List of Published SCAMPI Appraisal Results 


ORGANIZATION NAME: Lockheed Martin Corporation 
SPONSOR NAME: Valter VVinkler 
LEAD APPRAISER NAME: Eugene Miluk 
SEI PARTNER: Software Engineering Institute 
APPRAISAL END DATE: 10/14/2005 
MATURITY LEVEL ASSIGNED: 5 
APPRAISED ORGANIZATIONAL UNIT: 

Entity Name: Integrated Systems & Solutions 

Location(s): King of Prussia, PA 
CMMI MODEL USED: CMMI-FSE/SVWIPPD/SS, V1.1, Staged 
APPRAISAL METHOD USED: SCAMPI v1.1 


MODEL SCOPE & CAPABILITY RATINGS ASSIGNED: 


Process Management Project Management Engineering Support 
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What is Benchmarking? 


Term 
Benchmark To take a measurement against a reference point. 
Benchmarking A process of comparing and measuring an 


organization with business leaders anywhere in 
the world to gain information which will help the 
organization take action to improve its 
performance. 


The Benchmarking Management Guide 
American Productivity and Quality Center 
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Types of Process Benchmarking 


Internal studies Compare similar operations within different units 
of an organization. 


Competitive Target specific products, processes, or methods 
Studies used by an organization’s direct competitors. 
Functional or Compare similar functions within the same 


industry studies —_ broad industry or compare organizational 
performance with that of industry leaders. 


Generic Compares work practices or processes that are 
benchmarking independent of industry. 
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The Performance Benchmarking Consortium 


During April 2006, SEI launched a vendor and industry collaboration on 
benchmarking software project performance. 


¢ Provide tools and credible data for goal-setting and performance improvement 


¢ To combine benchmark data from multiple repository sources thereby creating 
a superset of information for benchmark and/or performance comparison 


¢ Establish specifications for the collection and comparison of 
benchmark data from different vendor sources 


¢ Allow companies to leverage existing data to help them establish 
and achieve their business goals 
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Current PBC Members as of October 2006 


Consortium members are leaders in software measurement and 
benchmarking from consultancies, industry, and academia. 


David Consulting Group PRIM 

Galorath Incorporated QSM 

ISBSG Raytheon 

Lockheed Martin SE! 

Motorola SPR 

Oracle Sie 

PRICE Systems University of Ottawa 


* International Software Benchmarking Standards Group (ISBSG) 
** Software Technology Transfer Finland (STTF) 
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Motivation 


Organizations want a way to gauge their performance and to compare their 
performance with others in their industry. 


Data on project performance is needed to demonsirate the impact of process 
improvement. 


Benchmarks 


¢ provide a reference point for interpreting performance 


¢ facilitate interpretation by setting specifications Cost Quality 
for how performance measurements 
are collected 
Customer 


Schedule : : 
satisfaction 


Performance Benchmarking Consortium 


== Software Engineering Institute CarnegieMellon —_ mark kasunic, 11-15-2008 


© 2006 Carnegie Mellon University 


Is There Community Interest? 


During June, 2006, the PBC conducted an initial survey to assess the voice of the 


customer. 
can authorize sharing of assigns a high degree of value to 
organization’s performance & software project performance 
data benchmarks 
12 out of 14 
or 
85.7% 
part of a team who can assigns a high degree of value to 
authorize sharing of & software project performance 
organization’s performance benchmarks 
data 
57 out of 63 
or Results of 2006 PBC Survey 
90. 5% * Sample size = 800 


Response Outcome = 25% 
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Broader Data Provided by PBC 


Vendor 
1 


Vendor Vendor 
2 3 


Vendor 
5 


Vendor 
4 
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Broader Data Set Provided by PBC 


Coeeeeesseeeeeseseeeseesece 


“* Member #1 
“* Member #2 
“* Member #3 
* Member #4 
* Member #5 
“* Member #6 
* Member #7 


“¢ Member #n 
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PBC Provides Broader Coverage of 
Application Domains 


™ 


Nttpo:/wa 


Education 


Insurance 
Health and Medicine 


Military Systems 
Financial & Banking 
Manufacturing 
Electronics 


_. Communications 
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Presentation Topics 


Introduction 
¢ Overview — The Process Benchmarking Consortium 
¢ Motivation & Benefits 
=~ Ground covered 
PBC Working Teams 
¢ Concept of operations 
¢ Voice of customer 
¢ Specification 
Next Steps 
¢ Near-term goals 
¢ How you can participate 
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Ground Covered - 1 


Kick-Off Workshop at SEI [April 19-20, 2006] 


¢ 14 presentations by workshop attendees; Discussion of current 
enchmarking issues and ways to address them. 


¢ Brainstorming issues on how to proceed. 


¢ Initiative to conduct survey to obtain community input on factors most 
likely to affect software project performance. 


¢ Performance Benchmarking Consortium iS born. 
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Ground Covered - 2 


PBC Workshop [June 28-29, 2006] 


¢ Initial survey addressing performance factors - results presented 
¢ Planning and goal-setting 

¢ Soliciting member input on the PBC business concept 

¢ Initial concepts about PBC products and services 


¢ Setting up work teams 
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Ground Covered - 3 


PBC Workshop [October 4-5, 2006] 
¢ Work team status reports and planning 
¢ Discussion of ConOps (Concept of Operations) 
¢« Presentations by Member Companies 
¢« Discussion of PBC Measurement Specification 
¢ Selection of initial set of performance influence factors and 
performance indicators that will populate version 1 of the PBC 


repository. 


¢ PBC 2006 planning and 2007 goal-setting 
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PBC Member Alignment is Happening 


Work during 2006 as focused on 


¢ Developing a common 
understanding of the terrain 


¢ Developing common goals 
¢ Cultivating a shared commitment 


October, 2006 


April, 2006 
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Current PBC Teams 


\ K ', ConOps Team ee Nn eT eee Se EL 


Voice of Customer Team rrr 


Specification TCAM | 


Pilot Implementation Team 
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Presentation Topics 


Introduction 
¢ Overview — The Process Benchmarking Consortium 
¢ Motivation & Benefits 
¢ Ground covered 
PBC Working Teams 
=> Concept of Operations 
¢ Voice of Customer 
¢ Specification 
Next Steps 
¢ Near-Term Goals 
¢ How you can participate 
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Concept of Operations - 1 


Member Assets ‘ Subscriber Performance Data 


PBC Member Companies | Subscriber Organizations 


, : | ~ Customized reports 
Members provide 
services & reports to : 


their internal or Measurement 
external customers _ Specification 
using PBC Assets in ee ee eee ee 
addition to, or in oi 2 
combination with, PBC Performance Reports 


their own 
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Concept of Operations - 2 


PBC Member Assets PBC 
=a | Measurement 4 Subscriber Organizations 
Specification . m 


Using specification allows 
subscribers to make valid 
comparisons. 


Becomes an ISO standard? 


Subscriber Inputs 
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Concept of Operations - 3 
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=~ Voice of Customer 
¢ Specification 
Next Steps 
¢ Near-Term Goals 
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Voice of Customer Team 


This team is focused on ensuring that the voice of the customer is 
represented as PBC products are planned for development. 


¢ Conducted a survey to obtain community input about factors that 
influence software project performance 


¢ Planning a birds-of-feather session for next SEPG Conference to 
solicit community input to influence the development of 


PBC products and services 
I | } 
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Specification Team Work 


The Spec Team is working to develop common definitions. 


Performance Influence Factors Performance Indicators 

¢ Process maturity ¢ Defect density 

¢ Application domain 

¢ Stability of requirements ¢ Schedule predictability 
¢ Size ¢ Productivity index 

¢ Project type ¢ Project delivery rate 


« Team size 


¢ Developers’ functional knowledge 
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Specification Team — Building Common 
Definitions 


S/W Project A 
Meas. Definition A 


S/W Project B 
Meas. Definition B 


me) eral amnycele meso) an] oy-lk= ie 


S/W Project C 
Meas. Definition C 
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S/W Project D 
Meas. Definition D 
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Specification Team — Building Common 
Definitions 


S/W Project A 
Meas. Definition A 


S/W Project B 
Meas. Definition B 


S/W Project C 
Meas. Definition C 


S/W Project D 


Meas. Definition D a 
The Specification Team 
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Specification Team — Building Common 
Definitions 


A cool, refreshing common 
definition for “time-to-market” 
that permits valid 
comparisons between 
software projects 
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PBC Measurement Specification 


Subscriber Organizations 


Using specification allows 
subscribers to make valid 
comparisons. 


Becomes an ISO standard? 


Subscriber Inputs 
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¢ Overview — The Process Benchmarking Consortium 
¢ Motivation & Benefits 
¢ Ground covered 

PBC Working Teams 
¢ Concept of Operations 
¢ Voice of Customer 
¢ Specification 
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Near-Term Goals 


The PBC will 
¢ create a set of process specifications for the consistent and meaningful 
collection, analysis, and dissemination of comparative performance 
benchmarks for software projects 
¢ develop a data model that will facilitate 
the aggregation and comparison of data 
from different sources 


¢ pilot test solution components 


¢ develop version 1.0 of the PBC data repository 
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Are You Interested? 


“* The PBC will be expanding membership during 2007. 


** Individuals who become members during 2007 will participate in 
proof-of-concept trials and provide feedback on the PBC Concept of 
Operations. 


“* If your organization is interested in sponsoring you as a member or if 
you are interested in becoming a subscriber of future PBC products 
and services, then please send email to Mark at 


mkasunic@sei.cmu.edu 


** Individuals who contact us will be added to our database. We will 
periodically send email about progress and future plans for the PBC. 


Thank-you 
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Why Do You Need a Maturity 
Level 5 Supplier? 
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November 2006 
Denver, Colorado 
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Agenda 


@ Myths 
@ Characteristics of ML2 — ML5 Organizations 


@Buyer’s Responsibility for Defining the System 
Up Front 


@Buyer’s Responsibility for Managing the 
Supplier 


@ Supplier Relationship 
@ Supplier Selection Criteria 


@ Continuous Representation, Capability Levels 
and Supplier Capability Evaluation 
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Myths 
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Myths 


You must “trust” your supplier or you should not 
choose them 


You must never allow your supplier to suggest a 
change to the requirements or to the contract 


If you do not have so much expertise, it is quite 
OK to have your Supplier to define your 
requirements for you and the Acceptance 
Testing criteria 


It is also acceptable for the Supplier to conduct the 
Acceptance Testing for you 


@ Using a Supplier is always cheaper than | 
spi Oping a product or product component in- 
ouse 
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Myths - 2 


If you choose a Maturity Level 5 Supplier, it 
greatly reduces the work for the Project, 
because the Supplier will “manage itself” 


If you need multiple suppliers, ML 5 Suppliers 
will be more willing to cooperate to Support your 
business objectives 


@ML 5 Suppliers are able to demonstrate 
expected performance regardless of the 
application domain 


ML 5 Suppliers are able to support the culture 
of any Buyer they are supporting 
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Characteristics of 
ML 2 - ML5 Organizations 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 7 


Maturity Level 2 
Characteristics 


@ Projects of the organization have ensured that 
processes are planned and executed in accordance 
with policy 


@ Projects employ skilled people 


@ Adequate resources are made available to the project 
team members 


@ Controlled outputs are produced 


@ Relevant stakeholders are involved, monitored, 
controlled, and reviewed and are evaluated for 
adherence to their process descriptions 


@ Status of the work products and the delivery of services 
are visible to management at defined points 


@ Work products are appropriately controlled 
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Maturity Level 3 
Characteristics 


@ Processes are well characterized and 


understood, and are described in standards, 
procedures, tools, and methods 


These standard processes are used to establish 
consistency across the organization 


The standards, process descriptions, and 
procedures for a project are tailored from the ~ 
organization’s set of standard processes to suit a 


particular project or organizational unit and therefore 
are more consistent 


Processes are managed more proactively using 
an understanding of the interrelationships of the 

process activities and detailed measures of the 

process, its work products, and its services 
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Maturity Level 4 
Characteristics 


@The organization and projects establish 
quantitative objectives for quality and process 
performance and use them as criteria in 
managing processes 


@ Quantitative objectives are based on the needs 
of the customer, end users, organization, and 
process implementers 


@ Quality and process performance is understood 
in statistical terms and is managed throughout 
the life of the processes 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 10 


Maturity Level 4 
Characteristics - 2 


For selected subprocesses, detailed measures 
of process performance are collected and 
Statistically analyzed 


@ Special causes of process variation are 
identified and, where appropriate, the sources 
of special causes are corrected to prevent 
future occurrences 


@ At maturity level 4, the performance of 
processes Is controlled using statistical and 
other quantitative techniques, and is 
quantitatively predictable 
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Maturity Level 5 
Characteristics 


@ At maturity level 5, an organization continually 
improves its processes based on a quantitative 
understanding of the common causes of 
variation inherent in processes 


@ Quantitative process improvement objectives 
for the organization are established, continually 
revised to reflect changing business objectives, 
and used as criteria in managing process 
improvement 


Both the defined processes and the 
organization’s set of standard processes are 
targets of measurable improvement activities 
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Deciding To Use A 
Supplier 
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Buyer’s Responsibility for 
Defining the System Up Front 
a 
@ Requirements 

© Business 

© Technical 

& Performance 

© Quality 

& Competition 

& Competence 

& Acceptance Criteria 


@ WBS > SOW 

@ Initial Planning 

@ Critical Path 

@ Initial Risk Identification 
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Buyer’s Responsibility for 
Defining the System Up Front - 2 


Se aaa eee eee 
@ Initial Technical Solution 


@ Deliverables 

@ Supplier Relationship 

@ Selecting the Supplier 

Establishing the Supplier Agreement 

@ Performing a Kick-Off with the Supplier's Project Team 
@ Monitoring the Supplier's Performance 


@ Evaluating Selected Supplier Processes and Work 
Products 


@ Transferring the Product or Product Component from 
Supplier to Buyer environment 
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Requirements 
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Deciding to Use a Supplier 
Requirements 


@ Understand the “customer’s” requirements 


Must be at a low enough level to be able to 
decide which of its customer requirements the 
Buyer wants the Supplier to be responsible to 
implement 
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Deciding to Use a Supplier 
Requirements - 2 


am 
@ Categories of questions for consideration 


& How are those requirements related to the Buyer’s 
organizational business requirements? 


& What is the Buyer’s relationship with the Supplier? 

& What technical ability must the Supplier have? 

& What technical performance is required and is the supplier 
capable? 


¢ What other projects can the Supplier discuss and show the 
actual performance delivered? 


& What quality factors is the Buyer seeking for the Supplier to 
deliver: 
* Reliability 
* Maintainability and Expandability 
* Interoperability 


@ See Template for Deciding to Use a Supplier 
> Template to Help Decide to Use a Supplier.doc 
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Identify System Requirements 


@ Determine goal and scope of the effort 
Review customer input 


@ Gather high-level Requirements 
Identify the goals and requirements of project 
Describe current state or system 
Describe change to system or new system 


Document justification for system change or new 
system 


Identify security requirements 
Identify computer resource requirements 
Identify internal and external interfaces 
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Review Customer Input 


@ Overview 

@ Objectives 

@ Assumptions and Constraints 
@ Requirements 


Functional requirements 

* what the system should do, not how 
&Non-functional requirements 

* constraints on the system 
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Describe System Objectives 


@ Reason for the system 
Why is it being built? 
* for Research and Development 
* for a specific customer 


* to meet existing/anticipated market 
needs/requirements 


* to upgrade an existing system to newer 
technology 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 21 


Describe System Objectives - 2 


@ Reason for the system 


What is the problem it is trying to solve? 
* a known problem 


* an anticipated problem (e.g., loss of 
maintenance/production support for existing 
technology) 
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Develop System Overview 


@The overview describes the interactions 
between the system and its operational 
environment 


Other systems with which it must interface 
Protocols it must use to interface with them 


Who the users are and in what manner they will 
use it 


Expected change in its environment over its 
lifetime 
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Document Your Assumptions 


@ All assumptions should be documented 
Captures thinking at the time 


Easier to find where wrong assumptions were made 
weeks and months later 


Reduces the risk of finger pointing about who made 
the right or wrong assumption 


Necessary for backup or someone who may take 
over the role of the person who worked on the 
decision to use a Supplier “6 
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List System Constraints 


@ Performance @ Development standards 
@ Efficiency @ Libraries 
@ Response times @ Operating Environment: 
@ Capacities & System compatibility 
© ately Te eeietee (Hane arel 
@ Security software) 
@ Quality issues © Hardware 
© Maintainability © Operating system 
© Portability 
© Reliability 
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Acceptance Criteria 


@ Acceptance Criteria (the criteria that a product or 
product component must satisfy to be accepted by a 
user, customer or authorized entity) should be part of 
the requirements capture and specification process 


Who will perform the acceptance testing? 


What environment or portion of the user’s environment 
must be exercised to satisfy the acceptance criteria? 


How much simulation will be allowed? 
What process will be followed if errors are found? 


¢ What classification of errors must be fixed before the 
system will be accepted? 


¢ What classification of errors may allow the system to 
be accepted in the event that workarounds can be 
provided? 
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Work Breakdown 
Structure 
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Determine Resources 
Needed 


@ Develop initial high-level work breakdown 
structure (WBS) PP-SAM-Starterkit- 


WBSExampleContentsv2.0.doc 
@ Make initial high-level rough estimates 
Size & Complexity 
© Effort 
Critical computer resources 


Identify all participant roles 


Wd 
8 og 
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Determine Resources 
Needed - 2 


@ Develop high-level staffing profile including 
need for contracting support 


@ Determine training needs 


@ Identify the QA activities to be performed and 
the required support 


@ Determine CM activities and support 
@ Determine the Test activities and support 


@ Identify groups external to the project for inter- 
group coordination 


\ ! 


@ Develop preliminary schedule 
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Initial Planning 
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Initial Estimation 


Provide initial estimates for the following 
Size 
© Effort 
Cost 
Schedule 
Risks 
Computer Resources 
Facilities and Test Environment 
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Resources Required 


What must one have to get this project done? 
@ Staff 
@Tools 
Analysis 
Design 
Project Tracking 
©CM 
Testing 
Facilities 
@ Systems 
@ Training 
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Staff Requirements 


@ Project Manager 


Detailed definition of skills and experience 
needed 


@ Detailed definition of time-frame needed 


Staff Requirements for planning and 
implementation phases of the project 


AT 
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Resource Planning 


People 


“Person months and people 
are not interchangeable.” 
“You cannot complete the 
project in half the time by 
doubling the people.” 


Frederick Brooks 
The Mythical Man-Month 


© 2006 Kasse Initiatives, LLC 


Staff Loading Curves 


Alternative Staffing Strategies 
to Accelerate the Schedule 


Z 4 6 8 10 12 14 
“Impossible Region” Maximum Schedule 


Acceleration £25% 
(shorten or extend by 25%) 


Barry Boehm 
Software Engineering Economics 
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Critical Path 
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Create an 
Initial Schedule 
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Critical Path 


@ Determine Tasks Dependencies 
Calculate the Critical Path (CPM ) schedule 
@ Integrate Resources into schedule 
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Microsoft Project Gantt Chart 


February 4 February 11 


28 | _Februa | 
Task Name Duration 
—eTTT 


1 Develop Software System 17d 


LS 
1.1 Design Phase 6d a 
1.1.1 Develop Funct 2d a 
1.1.2 Complete Desi 2d 
1.1.3 Develop Accer 2d 
1.2 Implementation Pha 8d 
1.2.1 Complete Deta 2d a 
1.2.2 Generate Code Ad 
1.2.3 Document Sys 3d —_—— 
10 1.2.4 Develop ATP 2d 
11 1.2.5 Develop Site P 1d 
12 1.2.6 Complete Test 2d = 
13 1.3 Installation Phase 3d 
14 1.3.1 Install System 1d 
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Ss) 


Oo; CIN] Oo] oT] BI] ®|] NY] FR 


RISksS 
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Risk Definition 


@ Any event that could impact project results 
Effort and Cost 
Schedule 
Functionality 
Performance 
Quality 
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Technical Solution 
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Recommend Strategy Tasks 


@ Develop high-level development strategy 
@ Develop high-level system architecture 


@ identify proposed language/development 
environment 


Identify potential use of tools 
Identify potential use of COTS 
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Recommend Strategy Tasks - 2 


@ Describe potential alternative solutions 


@ Describe advantages and disadvantages of 
each potential solution in order to be in the best 
position to evaluate the Supplier’s 
recommendations 
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Deliverables 
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Review Preliminary 
Project Plan 
eee 
@ Prepare preliminary project plan 


@ Ensure that all deliverables are clearly specified; 


& Product components in software and hardware integrated into a 
system 


& Architecture Specification 

Design Specifications 

© Installation Manual 

& Operator’s Manual 

© User’s Manual 

& Functional Configuration Audit Report 
& Physical Configuration Audit Report 
& Version Description Document 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 45 


Supplier Relationship 
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Supplier Relationship 


Requirements Trust 


Supplier — oe 
, = _ (Xe) | Feleleyestceys 


eee | i ao eee 
yl CE 7 arn a ¢ 7 Supplier as proven 
Supplier | IS , Supplier has proven Rihana rites dl 
cee Sey le aby tO Peon , | inellah and is willing to ; 
, set of , ~withinthe agreedupon (7 ; ge 
requirement / time and budget eee 
; eC SEN ESSERE NN CES yl entrepreneurial risk 


X~ 


\SLLLLLLILIELLLLLLLLLLLLLILELLLILLLELEEELILEEEELLELELLLEL IEEE 
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Supplier Selection Criteria 
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Supplier 
Evaluation Criteria 


@ Suppliers are selected based on their ability to perform 
the work according to predefined evaluation criteria 


& Prior documented performance in similar applications 
& Geographic location 

& System Engineering capabilities 

& Software Engineering capabilities 


& Knowledge, skills, and numbers of staff available to perform the 
work 


Available resources (facilities, hardware, software, training) 
& Capability evaluation (Appraisal Results) 


© KI Supplier Evaluation Checklist >Consolidated Supplier 
Evaluation Criteria - v2.7 Categories.rtf 
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Establish the Supplier 
Agreement 
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Establish the 
Supplier Agreement 


Establishing and maintaining the supplier 
agreement provides the supplier with the project 
needs, expectations, and measures of 
effectiveness 


@The supplier agreement typically includes: 
Statement of work 
Terms and conditions 
List of deliverables, schedule, and budget 


Defined acceptance process including acceptance 
criteria 
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Establish the 
Supplier Agreement - 2 


Identification of project and supplier representatives 
responsible and authorized to agree to changes to 
the supplier agreement 


© Identifying the process for handling requirements 
change requests from either side 


Identifying processes, procedures, guidelines, 
methods, templates, etc., that will be followed 


© Identifying critical dependencies between the project 
and the supplier 


Identifying the form, frequency, and depth of project 
oversight the supplier can expect from the project 


* includes evaluation criteria to be used in 
monitoring the supplier’s performance 
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Establish the 
Supplier Agreement - 3 


Identifying the supplier’s responsibilities for 
ongoing maintenance and support of the acquired 
products 


Identifying warranty, ownership, and usage rights 
for the acquired products 
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Monitor the Supplier’s 
Performance 
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Monitoring Performance 
Activities 


Progress/Status Formal 
Reviews Milestone 


Project Plan : Product 

and Other A ccite !) Acceptance & 
echnica : 

Approvals Project 


[pp | Ra | ap | pp [our] i [ st | Pa | cs” 
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Monitoring Concerns 


Computer 


mre ay 


it 


| Dg 


Quality » ‘ 


Management 


Issues 
and 
Action 


Risk Items 
and 
Mitigation 
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Supplier Quality Assurance 
Monitoring 


@The project's Quality Assurance representative 
monitors the supplier’s quality assurance 
activities 

The supplier's plans, resources, procedures, and 
standards, for quality assurance are periodically 


reviewed to ensure they are adequate to monitor the 
supplier's own performance 
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Supplier Configuration 
Management Monitoring 


@The project’s Configuration Management 
representatives monitor the supplier's 
configuration management activities 


The project and the supplier coordinate their 
activities on matters relating to configuration 
management to ensure that the supplier’s products 
can be readily integrated or incorporated into the 
project environment 
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Monitoring Selected 
Processes and Evaluating 
Selected Work Products 
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Monitor Selected 
Supplier Processes 


@ In situations where there must be tight alignment 
between some of the processes implemented by the 
supplier and those of the project, monitoring these 
processes will help prevent interface problems 


@ The selection must consider the impact of the supplier's 
processes on the project. 


On larger projects with significant subcontracts for development 
of critical components, monitoring of key processes is expected 


On smaller, less critical components, the selection process may 
determine that monitoring is not appropriate 


© The overall risk should be considered in selecting processes to 
be monitored between these extremes 
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Monitor Selected 
Supplier Processes - 2 


@ The processes selected for monitoring should include: 
& Engineering 
& Project management 


& Support processes critical to successful project performance 
such as quality assurance and configuration management 


@ There should be sufficient monitoring to detect issues, 
as early as possible, that may affect the supplier's ability 
to satisfy the requirements of the supplier agreement 


@ Analyzing selected processes involves taking the data 
obtained from monitoring selected supplier processes 
and analyzing it to determine whether there are serious 
issues 
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Measurement and Analysis 
Monitoring 


@ Measurement and analysis of the product 
components provided by suppliers is essential 
for effective management of the quality and 
costs of the project 


@ It is possible, with careful management of 
supplier agreements, to provide insight into the 
data that support supplier-performance analysis 
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Evaluate Selected Supplier 
Work Products 


@ The scope of this specific practice is limited to suppliers 
providing the project with custom-made products 


<& Custom-made products that present some risk to the program 
due to complexity or criticality should be examined as potential 
candidates 


@ Selected work products produced by the supplier should 
be evaluated to help detect issues as early as possible 
that may affect the supplier's ability to satisfy the 
requirements of the agreement 


@ The work products selected for evaluation should 
include critical products, product components, and work 
products that provide insight into quality issues as early 
as possible 
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Evaluate Selected Supplier 
Work Products - 2 


@Work products are evaluated to ensure the 
following: 


Derived requirements are traceable to higher level 
requirements 


The architecture is feasible and will satisfy future 
product growth and reuse needs 


Documentation that will be used to operate and to 
support the product is adequate 


Work products are consistent with one another 


Products and product components (e.g., custom- 
made, off-the-shelf, and customer-supplied products) 
can be integrated 
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Transitioning from the 
Supplier’s Environment to 
the Buyer’s Environment 
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Transitioning the Acquired 
Product From the Supplier to the 


Project 


@ Monitor the transition of the acquired products 
from the supplier to the project 


Ensure that the appropriate facilities to receive, 
store, use, and maintain the acquired products are 
available 


Ensure that the appropriate training is provided to 
assist in the transition 


Ensure that the storing, distributing, and use of the 
acquired products is performed according to the 
terms and conditions specified in the supplier 
agreement 


<7 en sOning Heim Supplier to Buyer 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 66 


Supplier Capability 
Evaluation 
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Focus is on quantitative |Causal Analysis and Resolution 


SOU MTT Fe ne Cee Organizational Innovation and Deployment 
improvement 


Process is measured Quantitative Project Management 
and controlled Organizational Process Performance 


. ____ {Requirements Development — Organizational Process Focus 
Process Is characterized /Technical Solution Organization Process Definition 


Seria Organizational Training 
Verification 
Validation Integrated Project Management 


Decision Analysis & Resolution Risk Management 


for the organization and |proquct Integration 


is proactive 


Process is characterized |Requirements Management Product and Process 
for projects and is often | Project Planning Quality Assurance 


eee Project Monitoring and Control Configuration Management 
Supplier Agreement Management Measurement & Analysis 
Process is unpredictable, 


poorly controlled, and 
reactive 


Capability Levels 


5 Optimizing 


4 Quantitatively Managed 


3 Defined 


2 Managed 


1 Performed 


O Incomplete 
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Supplier Capability Evaluation 


@ Capability Level 0 deals with Incomplete 
processes 


@ An incomplete process is a process that is 
either not performed or only performed partially 


One or more Specific Goals of the process are not 
performed 


@ Capability Level 1 deals with Performed 
processes 


A performed process is a process that 
accomplishes the work necessary to 
produce work products 
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Supplier Capability Evaluation - 2 


@ Capability Level 2 deals with Managed 
processes 


@A managed process is a performed (capability 
level 1) process that has the basic 
infrastructure in place to Support the process 
with the following characteristics: 


It is planned and executed in accordance with policy 


Employs skilled people who have adequate 
resources to produce controlled outputs 


Involves relevant stakeholders 
Is monitored, controlled, and reviewed 
Is evaluated for adherence to its process description 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 71 


Supplier Capability Evaluation - 3 

eee eee 

@ GP 2.1 Establish an Organizational Policy 

@ GP 2.2 Plan the Process 

@ GP 2.3 Provide Resources 

@ GP 2.4 Assign Responsibility 

@ GP 2.5 Train People 

@ GP 2.6 Manage Configurations 

@ GP 2.7 Identify and Involve Relevant Stakeholders 

@ GP 2.8 - Monitor and Control the Process 

@ GP 2.9 - Objectively Evaluate Adherence 


@ GP 2.10 Review Status with Higher-Level 
Management 


© 2006 Kasse Initiatives, LLC Version 3.0 Why Do You Need an ML 5 Supplier - 72 


Supplier Capability Evaluation - 4 


@GP 3.1 - Establish a Defined Process 
@GP 3.2 Collect Improvement Information 


@GP 4.1 Establish Quantitative Objectives for 
the Process 


@GP 4.2 Stabilize Subprocess Performance 


@GP 5.1 Ensure Process Improvement 
Objectives 


@GP 5.2 Correct Root Causes of Problems 
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Summary - Why Do You Need an 
ML5 Supplier? 


@ Deciding to use a supplier requires: 
Une cistanelng of the requirements to a low enough 
eve 


Developing the initial project planning 


Defining the expectations, resources, and constraints 
for a project that will make use of one or more 
Suppliers 


Determining the relationship the Buyer wants with 
the Supplier 


Developing the Statement of Work 
Developing the RFP 
Establishing the Evaluation Criteria 
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Summary - Why Do You Need an 
ML5 Supplier? - 2 


Evaluating the supplier’s organizational process 
capability against the requirements and constraints 


Evaluating the supplier’s quality and configuration 
management process capability 


Evaluating the supplier's management and 
engineering capability for the desired tasks 


Validating the supplier’s past project performance on 
projects of similar size, complexity, functionality, 
cost, schedule, quality, risk, and customer 
satisfaction 
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Thank You 
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CMMI Based Risk 
Management: 
Managing the Risk 
Attributed to Supplier 
Processes 


Presented By: 


Gary Sigmund, Program Integrator 
Defense Contract Management Agency 
Naval Sea Systems Operations 


CMMI Technology Conference an User Group 


15 November 2006 


DCMA — CMMI Based Risk Management 


| Defense Contract Management Agency | Contract | Defense Contract Management Agency | Agency 


Presentation Goals & Objectives 


o Defense Contract Management Agency (DCMA) surveillance 
activities can be effectively planned and implemented within 
the context of : 

> the CMMI 
> the contractor’s process capability and improvement strategy 


o DCMA surveillance can be used to identify the Risks which 
can be attributed to process; and monitor the effectiveness 
of contractor Risk Handling methods 


DCMA Topics to be Discussed 


| Defense Contract Management Agency | Contract | Defense Contract Management Agency | Agency 


The DCMA Risk Management and Surveillance Requirements 
o DOD 5000.2R 
o Risk Management Guide for DOD Acquisition - 6" Edition v1.0 
August 2006 
o Federal Acquisition Regulations 
The application of CMMI to DCMA Surveillance Activities 
o Monitoring of Supplier Risk Management Program 
o Contractor Process Improvement 
o Award Fee Decisions 
o Pre-Award Surveys 
The DCMA CMMI Based Surveillance Framework 
The transformation of CMMI Based Surveillance Data into 
customer desired Information 


DCMA “Process” - A critical contributor to program risk 


“To manage risk, the PMOs should focus on the critical areas 
that could affect the outcome of their programs. Work 
Breakdown Structure (WBS) product and process elements 
and engineering an manufacturing processes contain most of 
the significant risk events.” 


“Critical risk processes are the developer’s engineering and 
production processes which historically have caused the 
most difficulty during the development and/or production 
phases of acquisition programs.” 


“These processes include, but are not limited to, design, test, 
production facilities, logistics and management.” 


Risk Management Guide for DOD Acquisition 6th Edition, v1.0 August 2006 


The components of the CMMI offer the ideal suite of best practices to assist 
DCMaA with in-plant surveillance and risk monitoring. 


4 


DCMA Two conflicting perspectives on Acquisition Risk 
Defense Contract Management Af, EEE 


Government Risk: the 
probability/likelihood of acquiring a fully 
functional system on time, at the right price, 
with reasonable supportability and 
maintenance costs downstream. 


Contractor Risk: involves the inherent 


chance of making a profit or incurring a 
loss and acquiring future business. 


The Government PM and the Contractor may consequently have different 
views on specific Cost, Schedule, and Performance Risk levels and the 
amount of process which Is necessary 


fs) 


DCMA Risk Management and the CMMI 


| Defense Contract Management Agency | Contract | Defense Contract Management Agency | Agency 


oe 


Why should DCMA align the CMMI to DOD’s 
Supplier Risk Management and Surveillance 
Mission? 


DOD Agency Requirements: 
DCMA is the DOD component which performs Contract Administration 
Services in-plant; 

DCMA is required by Federal Law to administer contracts and perform 
Surveillance to assist the Government acquire systems which are 
developed, produced and managed from Contractor processes 


Contract Process Requirements: 
The contactor processes are key contributors to the risk of acquiring a 
System 


Contractor Process and Improvement Initiatives: 
The Contractor typically is modeling their processes and improvement 
Efforts to the CMMI architecture 


DCMA CMMI Based Risk Management 


ml are si investment returns? ; 
Program-Process Risk 


Information to be used by 


DOD DCMA, Contractor, and PMs 
POLICIES 
Process 
CMMI based Capability & 
FAR = Risk Performance 
monadement Indicators for 
Award Fee 


on 


Determination 


Real-Time Contractor 
Process Improvement 


DCMA_ CMMI Based Risk Management 


Defense Contract Management Agency 


What it is: 

o A framework for using the CMMI to perform Supplier Risk Management while performing 
surveillance required by the FAR. 

o A tool for exposing and handling the acquisition risk which can be attributed to contractor 
processes. 

o A5 Step Method to capture surveillance results, which are organized and analyzed by the 
components of the CMMI 

o A framework for linking and coordinating: 


> 
> 
> 
> 


— 


DCMA Process Reviews, Product Examinations, and Corrective Actions within the context of the CMMI 


Surveillance required by the FAR 


DOD Risk Management Policy and Guidance in a Post Award Environment 


DCMA activities within contractor’s process improvement strategies 


Why it’s CMMI Based: 


o Defacto Standard and Common Language for Government & Industry 


o Provides a suite of known “Best Practices” to focus surveillance efforts 


When it’s applied: 


o Used Daily 


DCMA CMMI Process Areas Mapping to the FAR 


Defense Contract Management Agency 


Mapping the CMMI Process Areas to the DCMA Contract Administration 
Service surveillance requirements mandated by the FAR 


FAR Subpart 42.3—Contract Administration Office Functions 


42.302 Contract administration functions. 

(a) The contracting officer normally delegates the following 
contract 

Aministration functions to a CAO: 


(40) Perform engineering surveillance to assess compliance with 
contractual terms for schedule, cost, and technical performance in the 
areas of design, development, and production. PP, PMC, RD, RM, TS, 
PI, VER, VAL, IPM, RM, CM, PPQA, M&A, SAM 


DCMA CMMI Process Areas Mapping to the FAR 


Defense Contract Management Agency 


Mapping the CMMI Process Areas to the DCMA Contract Administration 
Service surveillance requirements mandated by the FAR 


FAR Subpart 42.3—Contract Administration Office Functions 


42.302 Contract administration functions. 
(a) The contracting officer normally delegates the following 
contract Aministration functions to a CAO: 


(38) Ensure contractor compliance with contractual quality assurance 
requirements. PPQA 


DCMA CMMI Process Areas Mapping to the FAR 


Defense Contract Management Agency 


Mapping the CMMI Process Areas to the DCMA Contract Administration 
Service surveillance requirements mandated by the FAR 


FAR Subpart 42.3—Contract Administration Office Functions 


42.302 Contract administration functions. 
(a) The contracting officer normally delegates the following contract 
Aministration functions to a CAO: 


(41) Evaluate for adequacy and perform surveillance of contractor engineering 
efforts and management systems that relate to design, development, production, 
engineering changes, subcontractors, tests, management of engineering resources, 
reliability and maintainability, data control systems, configuration management, and 
independent research and development. PP, PMC,SAM, IPM, RSKM, REQM, RD, 
RM, TS, PI, VER, VAL 
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DCMA_ CMMI Based Risk Management 


Defense Contract Management Agency 


What are the Advantages of Using CMMI? vn 


> Validates Government expectations of 
Contractor to achieve/maintain appropriate 
levels of process capability and performance 

> DCMA is on-site Govt. Rep — Utilizing 
knowledge and past experience with day-to-day 
process and product performance. 

> DCMA surveillance is a continuous assessment 
of process performance - data always current —- 
differs from snap-shot appraisal methods 


DC CMMI - Based Risk Management Framework 


Step1 Plan & Prioritize Surveillance Step 2 Implement Surveillance 


PAST PERFORMANCE By ch tla ser 
a ae 


Process Reviews 
Product 


Examinations 
Data Collection 
Corrective Actions 


Record, Classify, and Step 3 STRENGTHS 


: . WEAKNESSES 
DCMA REPORT Categorize Surveillance Results 


[-In terms of COST, Process PS eReir 


SCHEDULE, TECHNICAL 
PERFORMANCE-] 


PRIORITY 


Step 4 


Step 5 Analyze & Map 
Report Surveillance and Surveillance 
Make Recommendations Results 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | 


Transformation of Data onsite 


Process Reviews 


Product 


Examinations DCMA CMMI Based 
Data Collection Observations 


Corrective Actions 


DCMA Surveillance Notes \ 
gathered during: Process 


Reviews, — 
Product Examinations, Data DCMA Findings — Issues - 
Concerns 


Analysis, 
Corrective Actions 


DCMA CMMI - Based Risk Management Framework 


Process Reviews 
Product Examinations 
Data Collection 
Corrective Actions 


Surveillance Data Collection Technique — 
Conversation & Discussion 


CMMI based data is collected while attending meetings and 
through day to day discussions of process with the contractor 


DCMA CMMI - Based Risk Management Framework 


Defense Contract Management Agency 


Surveillance Data Collection Technique — 


Process Reviews : : 
Review of Documentation 


Product Examinations 
Data Collection 
Corrective Actions How is documentation used by KTR? 


Define & Standardize process 

Indicate a Commitment to Use a process 
Provide evidence of process use 

Collect data about process performance 


Document Categories: 


Organization Level 
Project Level 
Implementation Level 


DC Translation of CMMI based Surveillance Data 
ramen into Risk related Information 


se Contract Management Agency 


Findings — Issues — Concerns are Used 
to: 


0 Identify Strengths & Weaknesses to 
the customer and the supplier 

o Adjust surveillance activities to 
handle the risk by working with the 
supplier to improve processes < 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | 


Transformation of Data onsite 


Process Reviews 


Product 


Examinations DCMA CMMI Based 
Data Collection Observations 


Corrective Actions 


DCMA Surveillance Notes \ 
gathered during: Process 


Reviews, — 
Product Examinations, Data DCMA Findings — Issues - 
Concerns 


Analysis, 
Corrective Actions 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | 


VE SG.2 Perform Peer Reviews 


STRENGTHS 
WEAKNESSES SDFs of TYZ_acad, and TYT_acad 
OF do not contain peer review records 
DEFECTS , ; 
— as required by project SSPM 


SQA and Project manager inputs to 
cost proposal did not include hours 
for SQA activities and /or 
Engineering activity to support peer 
reviews of design objects on three 
CSCls. 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | 


STRENGTHS 
WEAKNESSES 
or 
DEFECTS 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


VE SG.2 Perform Peer Reviews 


Design reviews are not being 
held on the fgU-PN4 program 


Software Development Plan, 
IMS, and Software Development 
Files do not reflect the planning 
for reviews of fgU-PN4 design 
objects 

DCMA review of SDFs reflect the 
absence of design review 
records as prescribed by XYZ's 
mandated process IDR_OO6A. 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | 


STRENGTHS 
WEAKNESSES 
or 
DEFECTS 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


VERIFICATION (VER) 


Defects are not being identified on 
three CSClIs within the Audio 
Conference Upgrade Project (VE SG 
3) 

Peer Reviews of Design Objects are 
not planned for three software 
configuration items on the Audio 
Conference Upgrade Project. (VE SG 
2) 


DCMA 


CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | Contract | Defense Contract Management Agency | Agency 


Analysis Step 1 summarize CMMI based 
Observations into Finding statements 
within each Process Area when 
applicable 


Analysis Step 2 generate a Risk related 
Issue or Concern which would be sent 
to the Customer and also warranta 
DCMA action to handle the risk. 


Analysis Step 3. generate Consequence / 
Impact Statements 

Analysis Step 4 generate a DCMA Risk 
Monitoring Strategy 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | 


DCMA Monthly Report to Customer 


Issue/Concern: XYZ Corporation's 
failure to conduct design peer 
reviews on three CSCI development 


oe a efforts may result in defects going 
\ <n undetected until the test phase. 
\ / Consequences / Impact: 
\ The test effort will likely be 
tow 
7. cx aa extended. 


An increase in development cost 
and a slippage in schedule 
should be expected if the 
developers are to rework and 
retest the fixes associated with 
the defects uncovered in the test 
phases. 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 


Product Examinations, Corrective 
Action Requests 
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DCMA Monthly Report to Customer 


Finding/Issue/Concern: XYZ 
Corporation's failure to conduct 
design peer reviews on three CSCI 
development efforts may result in 


<> 
oe _ defects going undetected until the 
\ test phase. 
N Consequences / Impact: 
The test effort will likely be 
wai 
CT 


extended. 

An increase in development cost 
and a slippage in schedule 
should be expected if the 
developers are to rework and 
retest the fixes associated with 
the defects uncovered in the test 
phases. 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 


Product Examinations, Corrective 
Action Requests 
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DCMA CMMI - Based Risk Management Framework 


Defense Contract Management Agency 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA Monthly Report to Customer 


DCMA Risk Handling and 
Monitoring Plan for next 30 days 


>Increase independent DCMA 
inspection of design products using 
contractor defined standards and 
conventions and/or tool set 

>Increase activity in planning efforts of 
follow-on efforts to ensure proposals 
include estimates for design peer 
reviews 

>Work with contractor SEPG to ensure 
absence of design peer reviews are 
reported by contractor’s SQA oversight 
function 


DC CMMI - Based Risk Management Framework 


Step1 Plan & Prioritize Surveillance Step 2 Implement Surveillance 


PAST PERFORMANCE Bees: oneTen 


Process Reviews 
Product 


Examinations 
Data Collection 
Corrective Actions 


Record, Classify, and Step 3 STRENGTHS 


: : WEAKNESSES 
Categorize Surveillance Results me 


DEFECTS 


DCMA REPORT 


[-In terms of COST, 
SCHEDULE,TECHNICAL 
PERFORMANCE-] 


Process 


Health Step 4 
Step 5 Analyze & Map 


Report Surveillance and Surveillance 
Make Recommendations Results 


PRIORITY 


SURVEILLANCE 
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DCMA CMMI - Based Risk Management Framework 
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CM SG.3 Establish Integrity; SG.2 Track 
and Control Changes: SG.1 Establish 


STRENGTHS 
Baselines 


WEAKNESSES 
or 


DEFECTS >» Audio Conference Enhancement Project 


SCM Plan was prepared during the 
proposal and updated 30 days after 
contract award --plan continues to be 
dynamic while under control of SCCB 

> SCCB Software/Documentation Change 
Requests identified in audit of the S/DCR 
database reflect activity against SCM Plan 
>Review of the SCM Plan’s revision history 
against S/DCRs reflect routine use and 
control of management plan 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 
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CM SG.3 Establish Integrity; SG.2 Track 
and Control Changes: SG.1 Establish 
STRENGTHS 


WEAKNESSES Baselines 


or “ 
DEFECTS - Audio Conference Enhancement 


Project SCM Plan Appendix IV 
includes a detailed list of software 
work products to be placed under 
formal configuration control; 

- list includes components of 


software development and support 
environment 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 
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Project Planning (PP) SG.3 Develop a 


Project Plan 


STRENGTHS ; : 
WEAKNESSES - Audio Conference Enhancement Project 


or SCM Plan Appendix IV includes a detailed 
DEFECTS list of software work products to be 
placed under formal configuration control 
(PP SP2.3); 


- List includes components of software 


development and support environment 
(PP SP2.3, CM SP1.1) 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 
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CM SG.3 Establish Integrity; SG.2 Track 
and Control Changes: SG.1 Establish 


TRENGTH 
= oe Baselines 


WEAKNESSES 
or 


DEFECTS -Customer approved changes initiated within 


Requirements Traceability Management (RTM) 
tool are directly linked to change control 
function of development documentation CM 
SP1.3-1, SP1.2-1; REQM SP1.3-1 


-Impact of change requests on developmentitest 
documentation and code is readily identified in 
computer generated reports CM SP1.2-1 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 
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STRENGTHS 
WEAKNESSES 
or 
DEFECTS 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


CONFIGURATION MANAGEMENT 


(CM) 


Audio Conference Upgrade 
Project planning activities 
include strong emphasis on the 
establishment of and 
compliance with a Software 
Configuration Management 
Plan (PP SP2.3); 


Management controls ensure 
regular use of activities 
outlined in the plan and the 
accuracy of the plan itself (PMC 
SP1.4) 


DCMA CMMI - Based Risk Management Framework 


| Defense Contract Management Agency | Contract | Defense Contract Management Agency | Agency 


Analysis Step 1 summarize CMMI based 
Observations into Finding statements 
within each Process Area when 
applicable 

Analysis Step 2. generate a Risk related 
Finding, Issue or Concern which would 
be sent to the Customer and also 
warrant a DCMA action to handle the 
risk. 


Analysis Step 3. generate Consequence / 
Impact Statements 


Analysis Step 4 generate a DCMA Risk 
Monitoring Strategy 


a2 


DCMA CMMI - Based Risk Management Framework 
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CONFIGURATION MANAGEMENT 
(CM) 


Audio Conference Upgrade Project SCM 
function provides routine access and 
rigorous identification and control over 
software work products throughout 
development life cycle 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 
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DCMA Data Sources: Proposal 

Evaluations, Process Reviews, 

Product Examinations, Corrective 
/ Action Requests 


DCMA Monthly Report to Customer 


Finding/Issues/Concern: XYZ 
Corporation's strong emphasis 
into the identification, status, and 
change control over work 
product configurations will likely 
enhance the customer/user’s 
ability to maintain integrity and 
traceability of system 
configuration throughout system 
life cycle. 
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DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA Monthly Report to Customer 


Consequences / Impact: 
Inconsistencies among work 
products caused by inadvertent or 
unauthorized changes will likely be 
uncovered and corrected throughout 
development life cycle. 


Increased visibility into product 
functionality and development 
progress will likely occur. 


The likelihood of cost overruns or 
schedule slippages occurring as a 
result of inconsistencies among work 
products should be reduced. 
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DCMA CMMI - Based Risk Management Framework 


DCMA Monthly Report to Customer 


DCMA Risk Handling and Monitoring 
Plan for next 30 days 

Reduce DCM independent 
auditing/oversight of project 
SCM function. Establish 
periodic spot-check validation of 
SQA audit function of SCM 
activity. 

Work with contractor SEPG and 
SQA to identify other 
opportunities for process 
improvement requiring DCM and 
contractor SQA insight. 


DCMA Data Sources: Proposal 
Evaluations, Process Reviews, 
Product Examinations, Corrective 
Action Requests 


DCMA CMMI - Based Risk Management Framework 


Defense Contract Management Agency 
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Purpose 


The purpose of this talk is to describe the major changes to 
the SCAMPI A appraisal method for v1.2. 
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Revision Process 


The CMMI Steering Group provided a long-term strategy 
and the upgrade criteria for v1.2. 


The SCAMPI Upgrade Team (SUT) reviewed change 
requests to identify possible changes for the v1.2 appraisal 
method documents: Appraisal Requirements for CMMI 
(ARC) and Method Definition Document (MDD). 


The CMMI Steering Group served as the configuration 
control board for v1.2 changes to the ARC and MDD. 


The SUT developed a draft of the ARC and MDD for review 
by lead appraisers in May 2006. 


The ARC and MDD were released as part of the v1.2 CMMI 
Product Suite. 
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SCAMPI Upgrade Team Members 


Mary Busby 
Jack Ferguson 
Sandra Cepeda 


Will Hayes 
James Hell 
Denise Kirkham 
Steve Masters 
Lisa Ming 
Charlie Ryan 
Beth Sumpter 
Ron Ulrich 

Joe Wickless 


Lockheed Martin (Co-chair) 
SEI (Co-chair) 

RD&E Command, Software Engineering 
Directorate 

SEl 

U.S. Army (In Memoriam) 
Boeing 

SEl 

BAE Systems 

SE! 

National Security Agency 
Northrop Grumman 

SEI 
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SCAMPI A V1.2 Major Themes 


Reduce complexity and ambiguity 

Provide additional guidance where needed 
Strengthen appraisal planning and conduct 
Strengthen appraisal reporting 

Define appraisal validity period 


Strengthen lead appraiser requirements 
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Reduce Complexity? 


The requirement for instruments (e.g., questionnaires) was 
removed. 


Only two types of objective evidence are now required: 
¢ documents 
e interviews 


The following sections in MDD were revised: 

e switched 2.2 Verify and Validate Objective Evidence and 
2.3 Document Objective Evidence so that the order of 
tasks reflects the natural order of conducting an 
appraisal 

e separated Verify Objective Evidence and Validate 
Preliminary Findings to better describe each process 
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Reduce Complexity 


The use of the term instantiation was changed: 


e Instantiation is now defined as “the implementation of a 
model practice used in the appropriate context within the 
boundaries of an organizational unit.” 


¢ The word “instantiation” for project and organizational- 
wide entities was replaced with “project” or “support 


group.” 


oS 
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Reduce Ambiguity 


The rating Not Rated was clarified: 


e Process areas outside of the model scope are rated as Out of 
Scope. For example, for a maturity level 3 appraisal, maturity 
level 4 and 5 process areas are rated as Out of Scope. 


e For process areas that have insufficient data to be rated, the 
rating is Not Rated. 

e Process areas in the model scope, but outside the 
organizational scope are rated as Not Applicable. The only 
CMMI-DEV process area that can be Not Applicable is SAM 
(as determined by the appraisal team). 

The practice characterization tables were revised: 
¢ clarified the use of virtual versus live interviews 
e changed “face-to-face interviews” to “oral interviews” 
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Provide Additional Guidance 


Guidance for readiness reviews was revised to include 
team and logistical readiness. 


Additional guidance was provided for using virtual methods 
(e.g., for interviews and briefings). 


Additional guidance was provided for using alternative 


practices (Appendix C: Alternative Practice Identification 
and Characterization Guidance). 
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Strengthen Appraisal Planning 
and Conduct 


Organizational unit sampling was revised to* 

e strengthen parameters and limits for organizational 
sampling (e.g., identifying a minimum number of focus 
projects) 

e include criteria for reporting sampling decisions 


The Conduct Appraisal Phase must now be complete within 
90 days. 


Appraisal team members are now required to sign final 
findings. 


* Changes to address sampling were extensive. Refer to the MDD for details. 
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Strengthen Appraisal Reporting 


The Appraisal Disclosure Statement (ADS) now requires the 
following information. 


Organizational sampling criteria and decisions (e.g., projects 
included, projects excluded, percentage of organization 
represented) 


Basis for maturity/capability level 4 and 5 appraisal results 
¢ subprocesses Statistically managed 
¢ mapping of these subprocesses to quality and process- 
performance objectives 


Signatures of both the lead appraiser and sponsor 
e The lead appraiser affirms that the appraisal scope is 
representative of the organizational unit. 
e The sponsor affirms the accuracy of the ADS and authorizes 
the SEI to conduct any audits deemed necessary. 
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Define Appraisal Validity Period 


V1.2 appraisal results are valid for a maximum of 3 years 
from the date of the ADS. 


V1.1 appraisals will expire on August 31, 2007 or 3 years 


after the date the appraisal was conducted, whichever is 
later. 
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Strengthen Lead Appraiser 
Requirements 


Prior to conducting a v1.2 SCAMPI appraisal, the following 
must occur: 
¢ Current candidate and authorized lead appraisers and 
team leaders must complete CMMI v1.2 Upgrade 
Training. 
¢ Candidate and authorized lead appraisers must attend 
SCAMPI Face-to-Face Training. 
¢ Those who want to conduct v1.2 SCAMPI level 4 or 5 
appraisals must be certified. Certification requirements 
address the following: 
- education, training, and experience in level 4 and 5 
concepts 
- completion of an oral exam 
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Summary 


The SCAMPI A appraisal method was revised based on 
change requests received to 

¢ reduce complexity and ambiguity 

¢ provide additional guidance where needed 

e strengthen appraisal planning and conduct 

e strengthen appraisal reporting 

¢ define the appraisal validity period 

e strengthen lead appraiser requirements 


The changes are intended to make appraisals more 
accurate, reliable, and efficient. 
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Selected CMMI® Data (FY06) 


17,630 Trained in Introduction to CMMI 


Net increase of: 
-108 Authorized Introduction to CMMI Instructors 
= 45 Authorized SCAMPI Lead Appraisers 
= 84 Authorized SCAMPI B/C Team Leads 


744 SCAMPI V1.1 Class A Appraisals conducted and 
reported to the SEI 


417 Publicly Reported SCAMPI V1.1 Class A 
Appraisals on the SEI Web Site 
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Number of CMMI Students Trained 
(Cumulative) 


30000 

25000 

20000 

15000 = CMMI (Staged) discont'd. 
12/31/05 

10000 & CMMI (Continuous) 
discont'd. 12/31/05 

5000 B CMMI (S&C Combined) 
0 


1999 2000 2001 2002 2003 2004 2005 YTD 
2006 


8-31-06 
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Transition from V1.1 to V1.2 Status - 10-12-06 


Introduction to CMMI Students 
e Registered for Upgrade Training — 464 
e Upgrades Complete — 190 


Lead Appraisers and Instructors 
e Registered for Upgrade Training — 561 


¢ Upgrade Training Complete — 256 
e Exams 
- Passed — 159 


- Failed (and not yet retaken) -— 7 
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CMMI Adoption Trends 


CMMI Web pages views in 
September 2006 

¢ 360K/month 

e Exceeded 24K/day 
The following were the most 
downloaded files on the SEI 


Web site in September 2006: 


¢ CMMI-DEV, V1.2 (PDF 
Version) 

¢ CMMI V1.1 Overview 
Presentation 

¢ CMMI V1.2 Overview 
Presentation 

¢ CMMI-DEV, V1.2 (Word 
Version) 


¢ CMMI V1.2 Model Changes 


Presentation 
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Average daily page views per quarter 
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CMMI Transition Status — 10/31/06 
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Training 
Introduction to CMMI — 58,178 trained 
Intermediate CMMI — 2,280 trained 
Introduction to CMMI Instructors — 449 
SCAMPI Lead Appraisers — 630 trained 
SCAMPI B&C-Only Team Lead -- 33 


Authorized 
Introduction to CMMI V1.1 Instructors — 404 
SCAMPI V1.1 Lead Appraisers — 451 
SCAMPI B&C Team Leads -- 432 
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Intro to the CMM and CMMI Attendees 
(Cumulative) 
60000 
50000 
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Number of SCAMPI vX Class A Appraisals Conducted by Year by Model 
Representation* 
Reported as of 31 August 2006 


*Where Representation is reported 


@ Staged 


® Continuous 
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Number of SCAMPI v1.1 Class A Appraisals Conducted by Quarter 
Reported as of 31 August 2006 
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Combined Appraisal Opportunities 


Current ISO 9001 (Combined ISO Surveillance 
SCAMPI ‘A’ using Cat ‘C’ appraisal) 
ISO 9001 z = 


IA 
ISO 9001 


... continues to 
demonstrate 
compliance with 
ISO 9001:2000 


Rating letter & or certificate 
with scope indicating 
“,,. Inaccordance with Level X” 


...no behaviours 
inconsistent with 
operating at level X 


Rating letter 
indicating level 
achieved 


Current CMMI 


(Cat ‘C’ appraisal) 


ane possible peat for assessment and surveillance 
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Adoption: What Else Is Happening? 


The Addison-Wesley SEI Series Book and: 
e A Guide to the CMMI 

CMMI: A Framework... 

CMMI Assessments 

CMMI Distilled: Second Edition 

CMMI SCAMPI Distilled 

CMMI: Un Itinéraire Fleche 

De kleine CMMI 

Interpreting the CMMI 

Making Process Improvement Work 

Practical Insight into CMMI 

Real Process Improvement Using the CMMI 

CMMI Survival Guide 


Systematic Process Improvement Using ISO 
9001:2000 and CMMI 


Balancing Agility and Discipline 
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How about SEI Publications? 


Technical notes and special reports: 

e Interpreting CMMI: 
- for Operational Organizations 
- for COTS Based Systems 
- for Service Organizations 
- for Marketing (in progress) 

¢ Using CMMI with: 
- Earned Value Management 
- Product Line Practices 
- Six Sigma 

¢ Supplementing CMMI for Safety Critical Development 
(in progress) 

e Demonstrating the Impact and Benefits of CMMI (and 
web pages — www.sei.cmu.edu/cmmi/results) 


—s 
_— 
-_——— 
= — 
——-— - 
——-- 


_— 


=— Carnegie Mellon 
—— Software Engineering Institute 


CMMI Today 


Version 1.1 CMMI Product Suite was released January 
2002. 


¢ CMMI Web site visits average over 12,000/day 
¢ Over 58,000 people have been trained 


¢ Almost 1600 “class A” appraisals have been 
reported to the SEl 


Now we want to continuously improve... 
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CMMI V1.2...and Beyond 
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Major Themes 


Reduce complexity & size 
Increase coverage 


Increase confidence in appraisal results 
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Reduced Model Complexity & Size 


Eliminated the concepts of advanced practices and 
common features 


Incorporated ISM into SAM; eliminated Supplier Sourcing 
(SS) addition 


Consolidated and simplified the IPPD material 
All definitions consolidated in the glossary 


Adopted a single book approach (i.e., will no longer 
provide separate development models) 


Report size reduced 15% from either predecessor; PAs 
reduced 12% 
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Supplier Agreement Management 


Specific Goal Specific Practice 
Establish Supplier 1.1 - Determine Acquisition Type 
Agreements 1.2 - Select Suppliers 


1.3 - Establish Supplier Agreements 


Satisfy Supplier 2.1 — Execute the Supplier Agreement 
Agreements 2.2 — Monitor Selected Supplier 
Processes 
2.3 — Evaluate Selected Supplier Work 
Products 


2.4 — Accept the Acquired Product 
2.5 — Transition Products 


v1.1 SP2.1 “Review COTS Products,” was eliminated. “Identify 
candidate COTS products that satisfy requirements” is a new 
subpractice under the Technical Solutions Process Area SP1.1, 
“Develop Alternative Solutions and Selection Criteria.” 
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Integrated Product and Process 
Development (IPPD) Changes 


IPPD material is being revised significantly. 


¢ Organization Environment for Integration PA removed 
and material moved to Organizational Process 
Definition (OPD) PA. 


e Integrated Teaming PA removed and material moved to 
Integrated Project Management (IPM) PA. 


e IPPD goals have been consolidated. 
- “Enable IPPD Management” in OPD 


- “Apply IPPD Principles” in IPM 


¢ Overall material condensed and revised to be more 
consistent with other PAs. 
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IPPD Changes 


V1.1 
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V1.2 
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SG3 = Apply 
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Increased Model Coverage 


Added hardware amplifications 


Added two work environment practices (i.e., one in OPD 
and one in IPM) 


Added goal and two practices in OPF to emphasize 
importance of project startup 


Updated notes (including examples) where appropriate so 
that they also address service development and 
acquisition of critical elements 


Updated name to CMMI for Development (CMMI-DEV) to 
reflect the expanded coverage 
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Added Hardware Amplifications and 
Examples 


Six hardware amplifications were created to add emphasis 
on hardware engineering. Here is an example from TS. 


SP 2.1 Design the Product or Product Component 
Develop a design for the product or product component. 


For Hardware Engineering 

Detailed design is focused on product development of 
electronic, mechanical, electro-optical, and other hardware 
products and their components. Electrical schematics and 
interconnection diagrams are developed, mechanical and 
optical assembly models are generated, and fabrication 
and assembly processes are developed. 


Hardware examples were also added to emphasize 
hardware engineering. 
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Added Work Environment Coverage 


Work environment standards are established at the 
organizational level in OPD. 


SP 1.6 Establish Work Environment Standards 
Establish and maintain work environment standards. 
The project’s work environment is established at the 
project level in IPM. 


SP 1.3 Establish the Project's Work Environment 
Establish and maintain the project’s work environment 
based on the organization’s work environment standards. 
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Organizational Process Focus 


V1.1 V1.2 
SG1 - Determine Process Improvement SG1 - Determine Process Improvement 
Opportunities Opportunities 


1.1 - Establish Organizational Process Needs 1.1 - Establish Organizational Process Needs 
1.2 - Appraise the Organization’s Processes 1.2 - Appraise the Organization’s Processes 


1.3 — Identify the Organization’s Process 1.3 — Identify the Organization’s Process 
Improvements Improvements 

SG2 - Plan and Implement Process SG2 - Plan and Implement Process Improvement 
Improvement Activities 2.1 - Establish Process Action Plans 

2.1 — Establish Process Action Plans 2.2 — Implement Process Action Plans 
2.2 — Implement Process Action Plans SG3 - Deploy Organizational Process Assets 

2.3 —- Deploy Organizational Process Assets and Incorporate Lessons Learned 

2.4 - Incorporate Process-Related Experiences 3.1 — Deploy Organizational Process Assets 

into the Organizational Process Assets 3.2 -- Deploy Standard Processes 


3.3 -- Monitor Implementation 


3.4 -- Incorporate Process Related Experiences 
into the Organizational Process Assets 
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Model Changes - Other 


Improved the Overview section (Part One) 
Improved clarity of how GPs are used 
e Moved generic goals and practices to Part Two 


e Added explanation of how process areas support the 
implementation of GPs 


e Added GP elaborations for GP 3.2 


Improved the glossary (e.g., higher level management, 
bidirectional traceability, subprocess) 


Limited the process areas that can be considered “not 
applicable” to SAM. 


Clarified material throughout the model based on over 1000 
change requests 


=> Carnegie Mellon 
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CMMI Model Combinations 
V1.1 V 1.2 


. 
Organizational Goal 
(OPD) 
Project Goal (IPM) 
Le 
Related 


Examples 


——_—— ap 
Supplier 
Sourcing 


ntegrated Product anc 
Process Development 


© 2006 by Carnegie Mellon University page 26 


=— Carnegie Mellon 
— Software Engineering Institute 


Model Structure? 


Continuous V1.1 Continuous V1.2 


Note: no base 
or advanced 
specific 
practices 
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Model Structure? 


Staged V1.1 Staged V1.2 


Common Features 


Note: no 
common 
features 
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CMMI V1.2 Schedules 


Version 1.2 CMMI Product 
Suite release 


Update material available 


August 25, 2006 


August 25, 2006 


Upgrade course available on- August 25, 2006 


line 


First Lead Appraiser “face to October 16, 2006 


face” 


V1.2 ADS required for all 
SCAMPIs 


Last V1.1 Intro training 


First expiration of V1.1 
appraisals (3 year validity) 


Last V1.1 appraisal 


October, 2006 


December, 2006 
August 25, 2007 


August 31, 2007 
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Beyond V1.2, 


Improved architecture will allow post-V1.2 expansion. 


e Extensions of the life cycle (Services, 
Outsourcing/Acquisition) could expand use ofa 
common organizational framework: 


- allows coverage of more of the enterprise or 
potential partnering organizations 


- adapts model features to fit non-developmental 
efforts (e.g., CMMI Services, CMMI Acquisition) 
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3 Complementary “Constellations” 
CMMI-Services 


provides guidance for 
those providing services 
within organizations and 

to external customers 


CMMI-ACQ 
«—/ provides guidance 
to enable 
informed and 
decisive 


acquisition 
leadership 
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Architecture & Constellations 


Development Amplifications 
Development Additions 
ePA XX 


MMI Framework 


eAcquisition Amplifications 
eAcquisition Addition 

*PA YY 

ePA XX 

«PA ACQ 


See BUBB Bee eee | teste ete laslnatestaeateaalelsleateatsleateanaleeialealetaleaalnaletalateatnbeleelnalnbalaateealetenantaleaataletalabetatenabalatslstatsanebalee sean telatalatatatetatet tell tttetet tet alal | 
/ 1. 7 


eServices Amplifications 
eServices Additions 

*PA ZZ 

“PA YY 

*PA SRV 
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Beyond V1.2, 


First two new “constellations,” CMMI for Services and 
CMMI for Acquisition, have been “commissioned” by 
CMMI Steering Group. Development will be in parallel with 
V1.2 effort; publication sequenced after V1.2 rollout. 


Northrop-Grumman Is leading industry group for CMMI 
Services. 


e Initial focus will be for organizations providing “DoD 
services” as well as internal IT: 


- System maintenance 
- Network Management, IT Services 
- IV&V 
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Beyond V1.2, 


SEI is coordinating development of CMMI-ACQ. 
¢ Will build upon General Motors IT Sourcing expansion 


e Will add government perspectives from both DoD and 
civil agencies 
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Initial CMMI-ACQ Key Acquisition Processes* 


Solicitation & 
Supplier 

Agreement 

Development 


Acquisition 
Management 


Acquisition 
Technical 
Solution 


Acquisition 
Validation 


Acquisition 
Requirements 
Development 


Acquisition 
Verification 


*based on initial CMMI-ACQ model developed by General Motors/SEI 


Carnegie Mellon 
Software Engineering Institute 


Planned Sequence of Models 


CMMI V1.1 ———— 


CMMI-DEV V1.2 
GM IT \ 
Sourcing CMMI-ACQ 


CMMI-AM Laer 
SA-CMM 
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Lean Process Design 
A Concept of Process Quality 


David N. Card 
dca@q-labs.com 


Q-Labs Shaping your Processes for Competitive Advantage 


A DNY Company 


Agenda 


Background and Objectives 
Concepts of Lean 

= Lean Process Design Process 
» Summary 


United Kingdom United States China Netherlands 
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Background 


= CMMI® requires the definition of processes 
that cover certain goals and practices 
¢ Requires “sufficiency” 
¢ Does not provide criteria for a “good” process — not 
an appraisal consideration 
= Lean principles provide “goodness” criteria for 
DrOCeSSeS 


= Lean usually applied as a re- engineering — 
technique, e.g., Kaizen 


CMMI® Is a registered trademark of Carnegie Mellon Univers 


France Germany Sweden United Kingdom United States China Netherlands 
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Objectives 


= Identify the process “goodness” criteria 
implicit in Lean principles 

= Explain how these can be applied during the 
design and initial definition of processes 


= Minimize later rework and re-engineering 


France Germany Sweden United Kingdom United States China Netherlands 
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The Lean Misconception 


= Lean Is not about “light weight” processes 


= “Lean” refers to reducing inventory and “work 
in progress” 
= Lean is accomplished through robust 
processes 
¢ Simple 
¢ Reliable 
e Standardized 
e Enforced 


Caveat: many flavors of Lean 


France Germany Sweden United Kingdom United States China Netherlands 
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Five Lean Principles 


= Value — identify what is really important to the 
customer and focus on that 


» Value Stream — ensure all activities are 
necessary and add value 


= Flow — strive for continuous processing 
through the value stream 


= Pull — drive production with demand 
Perfection — prevent defects and rework — 
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Views of Lean 


Industry 
nain 


= Five Observed Principles 
e Value 


e Value Stream 
e Flow 
e Pull 


e Perfection 
= Technical Practices 


e Similar to Six Sigma (including Statistical Procéss-Gentroly” 
¢ Adds queuing theory perspective 


{ Queuing )\ 
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Lean Techniques 


= Realize the principles of Lean 


¢ Methods and tools for defining, analyzing, and 
improving processes 


¢ Criteria for efficient and effective processes 
e Dimensions of process performance subject to 
control 
= Based on queuing theory (a process may be 
viewed as a system of queues) | 


Lean ts not just an attitude! 


rance rman en United Kingdom United States China Netherlands 


25 Fees; Shaping your Processes for Competitive Advantage 
A DNY Company 


Example Manufacturing Process 


Stock o 
| Assemble Wheels _ 
a Assemble Wheels Wheels 
Assemble Frames — 


Stock of\ ~__ 
Bikes 


Inventory and Work in Progress Enable 
Inefficient and Unreliable Processes! 


Assemble Bike 


China Netherlands 


United Kingdom United States 


France Germany Sweden 
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Example Software Process 


Software, ~___ 
Produc 


Test Software 


Waterfall model implies large WIP; 
Incremental and agile imply smaller WIP, 
but are they “robust” enough? 
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Limits to Performance 
Queuing Theory 


A DNY Company 


Continuous [Improvement 
r Increases Capacity and 
Reduces Variability 


: 
5 
C? = 0.25 
oo uM 02 os 04 ae 06 O7 06 39 10 
Capacity Utilization and Variability 
Variability —, Capacity 


Pa Utilization 


United Kingdom United States 


Netherlands 
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Implications of Queuing Theory 


= Broadens concerns for managing the 
Capacity of organizations 
= Leads to design criteria for process definition 


¢ Queues with certain properties operate more 
efficiently 


= Suggests candidate measures 
e Properties of queues are measurable 


United Kingdom United States China Netherlands 
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The Capacity Myth 


= Most engineering organizations behave as if 
their system/software development capacity 
is elastic 
e Capacity expands to accommodate the need 
e Projects are planned in isolation 


= Systems have limits to performance 


¢ Organizational performance must be managed to 
facilitate project Success 

¢ Organizational performance is not simply an jm 

aggregation of individual projects - 
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Lean Process Design 


= Why — begin the process improvement 
journey with efficient and effective 
processes, not just compliant processes 


= How — use Lean concepts to make 
decisions about the organization and 
composition of processes, more than 
human factors considerations 


= When — during process design and initial — 
definition, not just as later rework 
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Kaizen 


" A facilitated team activity 


= Focused on improving an existing 
process or (more commonly) subprocess 

= Applies Lean principles 

= Typically involves tools such as 

¢ Quality Function Deployment 

¢ Process Mapping 

e Value Stream Analysis 
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Process Size as a False Trail 


= Lean is not about the “size” or “volume” 
of the process definition 
= Process definition includes 
¢ Organizational standard process 
¢ Organizationally-defined alternatives 
¢ Project-defined alternatives 
= Volume of alternatives and variations 
often exceed the volume of the basic = 
organizational process — focus on these! — 
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Process Proliferation 


Eliminate OSP Organizationally Defined Alternatives 
Element 


Eliminate Project 


Projects Defined Processes Varistinne 
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Lean Process Design Process 


= Requires a focus on the architecture or 
“system design”, as well as the design of 
Subprocesses 


= Provides common and specific criteria 
for each level of design 

= Use estimates of process performance 
to help make design decisions 


= Incorporates traditional process 
definition conventions and notations 
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Lean Process Design Process 


Existing 
Process 


Needs 
Existing ISO 9001, 
Process CMMI, etc. 
Performance 


Performance 
Data 
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Lean Process Design Criteria 


Break work into “small” packages 
Process small packages of work continuously 
Distribute work evenly across subprocesses 
Minimize subprocess variations 

Minimize hand-offs between subprocesses 
Reuse subprocesses within the architecture 
Avoid redundant or duplicative tasks 
Eliminate “just in case” activities 
Minimize waiting and delays 

Identify and mitigate performance constraint: 
Understand downstream demand 


Za 
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Lean Subprocess Design Criteria 


= Use proven subprocesses 

= Ensure subprocess are followed 

=" Keep subprocesses simple 

= Plan for control of subprocess performance 

= Build quality assurance into each subprocess 

= Avoid redundant or duplicative tasks 

= Eliminate “just in case” activities 

= Minimize waiting and delays 

= Identify and mitigate performance constraints ZZ 
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Human Factors Criteria 


= Process definitions are intended for human 

users, must be 

e Understandable 

e Natural/Intuitive 

e« Accessible 

e Effectively supported by tools 

ae ml (Om 
= Process definitions are only one element of a 
process instantiation (also, people, tools, and 
input) 
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coftware’ 


Lean Software Management 
Case Study: Timberline Inc 


Queens University 
Belfast 


Used with Permission 


Peter Middleton 
Amy Flaxel 
Ammon Cookson 
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Work Balance Chart - Initial 


0 Engineering Design / Develop 


B QA Design Black Box Testing 


B QA Design & Execute White Box 
Testing 

@ Training Development 

& Tech Comm Development 

@ QA Smoke / Regression Testing 

> 

0 QA Develop & —S 

Automation _ a 


B QA Exect 


Estimated Cycle Time 


Middleton, Flaxel, Cookson 
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Work Balance Chart -— Post Adjustment 


0 Engineering Design / Develop 


AS 
‘, 


ua 
o 


0 Engineering Design / Develop 


ié%) 


B QA Design Black Box Testing 


~ 
o 


B QA Design & Execute White Box 
Testing 


a 
o 


& Tech Comm Development - 
Training Development (1/2) 


Estimated Cycle Time 
N 


- 


B QA Execute Black Box Testing - 
QA Smoke / Regression T 


Middleton, Flaxel, Cookson 
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Example: Cycle Time Analysis 


Run Chart 


Units-Work Type Cycle Times (Actuals) 


Middleton, Flaxel, Cookson 


United Kingdom United States China Netherlands 
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Performance Dimensions 


= Task size 

= Work arrival rate 
= Cycle time 

= Effort 

"= Delay/Waiting 

= Defects 

= Rework 
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Summary 


= Lean is not about the volume of process 
definitions 


= Processes can be designed, up front, to 
be more efficient and effective with Lean 
principles 

= Good process design requires 
engineering at both the system (overall 
process) and subprocess levels 


= Lean helps identify the critical measures 
of performance | 
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About Q-Labs 


= Consulting, Training and 


Appraisals in: 


@ Software Measurement 


= CMM/CMMI 


@ Lean/Six Sigma 


@ [SO 15504/SPICE 


m@ ISO (001 
@ 150 employees 
m ISO 9001 Certified 


France Germany Sweden 


A DNY Company 


@ A broad international client 
base, e.g. 
@ Alcatel, Bouygues Telecom, 


France Telecom, Orange 


AXA, BNP Paribas, Banques 
Populaires 


ABB, R. Bosch, EDF, IBM, 
Siemens, Schneider Electric, 
Thomson Detexis, Volvo, 
Sony 

Atomic Energy Board of 
Canada, FAA, Norwegian 
Ministry of Justice, Swedi: 


Army TAC OM 


United States China Netherlands 


Make Middle Managers the 
Process Owners 


~ 


CMMI Conference 
November 2006 
Denver, Colorado 


ran Welcome 


Tervetuloa itary 


MNAN hea 
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Agenda 


@ History of Assessment and Process 
Improvement 


@ Managing Process Change 
Building Support for Change 
<A Sample Change Model 
Attributes of Successful Changes 
Sponsors 
Middle Managers 
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Agenda - 2 


@Sample Process Improvement Infrastructure 
Senior Management Advisory Board 
Steering Committee 
Process Group 
Working Groups 


@ Middle Managers as Process Owners 
@ Summary 
@ Case Study (ING (Postbank)) 
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In the Beginning 
(Short History of 
Assessment and Process 
Improvement 
1988 - 1991) 


ra Quality Management 
Maturity Grid 

Measurement Stage I: Stage Il: Stage Ill: Stage IV: Stage V: 

Categories Uncertainty | Awakening | Enlightenment | Wisdom Certainty 


Management 
understanding 
and attitude 


Quality 
Organization 
Status 


Problem 
Handling 


Cost of Quality 
as % of sales 


Quali: 
Improvement 
Actions 


s ti f We don't know We know why 

vom an 9 why we have we do not have 
quality a problems with problems with 
quality quality 


Rantanss Fi “Quality is Free" by Phili tb 
© 2006 Kasse Initiatives, LLC He eondase oy ine ea euaear 
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History of Assessment and 
Process Improvement 


@ Organizations were encouraged to conduct an 
assessment 
Team was predominantly from the SEI 
* 5-6 SEI Assessment Team Members 
¢ 2-3 Organization Team Members 
SEI Assessment Team Leader led the assessment, 


and was responsible for the Draft and Final Finding 
presentation 


Final Report was written at the SEI based on the 
assessment results but only by SEI people 


Recommendations were developed by SEI 
assessment team members — approved by Watts 
Humphrey 
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History of Assessment and 
Process Improvement - 2 


@ Most of the organizations assessed in the early 
days of the SEI’s development of the Software 
Process Assessment method were ML 1 


Had problems with basic project management 
functions 


Most had difficulties writing a decent Project Plan 


The SEl’s assessment recommendations almost 
inevitably suggested that the organization develop a 
process improvement or “Action Plan” and carry it 


* OK, we cannot plan therefore the SEI suggests we 
make an Action Plan which is like a Project Plan 
for process improvement and implement it — Sure! 
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Managing Process 
Change 
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Managing Process Change 


@ There have been continuous advances in 
technology and process modeling: 


@ Technology 
Hardware capability and performance advances 
New user interfaces and application areas 
New tools 
New methods 
Web-based 
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Managing Process Change - 2 


@ Process Modeling: 
ISO 9000:2000 
Capability Maturity Model (CMM) 
EIA — 731 — Systems Engineering CMM 
Capability Maturity Model Integration (CMMI®) 
BOOTSTRAP 
ISO 15504 (SPICE) 
<EFQM (Quality Management Model) 
@ But even with of all of the process models and 
with our passion to incorporate technology to 


help us, we (the industry) seem to be getting 
worse, not better 


© 2006 Kasse Initiatives, LLC Version 2.2 CMMI - Mid Mgrs — Process Owners - 10 


Managing Process Change - 3 


@WHY? - Because we have not paid sufficient 
attention to process, people, and their reactions 
to change! 


4 
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Change 


@ Change is a pervasive aspect of our lives and 
almost a necessity for economic survival. 
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Process Adoption Curve 


Early “Never” 


Majority 


Late Majority 


Adopters 
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Principles of Process Change 


@ Major changes must be 
sponsored by Senior 
Management 


@ Focus on fixing the process, not 
assigning the blame 


@ Understand current process first 
@ Change is continuous 


@ Improvement requires 
investment 


@ Retaining improvement requires 
periodic reinforcement 
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Building Support for Change 


Legitimacy 


Support 


[courtesy JMaher] 
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Building Support for Change - 2 


= Legitimacy 


Support i 
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Organizations as Systems 


“Hea Outputs: 
Financial, Products 
Technological, Services 
Material, 
Resources 
input-output flow of materials, 
energy, information [Source: Morgan, 1986] 
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Managing Complex Change 
Requirements 


“Managing Technological Change” 
Carnegie Mellon University 
Software Engineering Institute 
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What People Need During 
Transition 


@ Information 

@ Inclusion 

@ Support 

Safety 

@ Skills 

@ Freedom from blame 
@ Rewards 


“Managing Technological Change” 
Carnegie Mellon University 
Software Engineering Institute 
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Attributes of Successful 
Changes 


@ Management Sponsorship 


@The change is consistent with the 
organization’s values, MISSION - 
and strategy 


@ Buy-in by those who will be 
affected 


@ Training 
@ Communication 
@ Reward system is changed 
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Change Effort Roles 


@ Inventors 

@ Entrepreneurs 
@ Integrators 
Experts 


@ Managers (Middle Managers) 
@ Sponsors (Senior Managers) 
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Sponsors /Senior Managers 


@ Provide top management credibility and support 
for change projects 


@ Are prime communicators in interpreting how 
the change fits into the overall culture and 
vision of the organization 


@ Focus on relating the change effort to the 
broader goals of the organization 


@ Communicate why the change is needed to the 
total organization 
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Middle Managers 


@ Middle Managers 


© Simplifies/interprets the direction 
provided by Senior Management 


Delegates responsibility to Project 
Managers and Leads 


© Assigns priorities based on 
organization’s business objectives 
and vision 


& Focuses on getting the job done at 
all costs 


© Serves as part of the Process 
Improvement Steering Group 


© Reports progress to the Sponsors or 
Senior Management including 
process improvement progress 
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Sample Process 
Improvement 
Infrastructure 
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Sample Improvement 
Infrastructure 


Development Organization 


Senior Management 
Middle Management 


Process Improvement Infrastructure 
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Senior Management 
Advisory Board (SMAB) 


@ Initially and continuously share the organization or 
business unit’s vision and business objectives 
© Helps individuals to realize the value of their efforts in their 


everyday work and the value of the process improvement 
effort 


@ Relates the change effort to the broader goals of the 
organization (business strategy) 


Sample Improvement 
Infrastructure 


Process Improvement Infrastruc Development Organization 


Senior Management | 


Middle Management 


© 2006 Kasse Initiatives, LLC 
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Senior Management 
Advisory Board (SMAB) - 2 


@ Looks for measurable improvements to support the 
business objectives 


@ Creates the environment to ensure process 
improvement becomes a part of the culture and vision of 
the organization 

<& Demonstrates regular and consistent visible support 
* for specific change actions 
¢ for continuous process improvement in general 


@ Communicates why the change is needed to the total 
organization 


& Approves/protects budget for the process improvement initiative 


@ Ensures that the necessary resources are available ina 
timely fashion 
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Steering Committee 


@ Ensures that process improvement activities are in 
line with the business goals as established by the 
SMAB 

& Reviews the proposed budget for the improvement effort 


& Makes recommendations to the SMAB regarding program 
direction, budget, and program risks 


Sample Improvement 
Infrastructure 


Process Improvement Infrastructure Development Organization 


Senior Management | 


Middle Management 


y 
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Steering Committee - 2 


@ Ensures that the necessary resources for the 
Working vines and Process Group are available 
in a timely fashion 


Establishes the WGs for specific purposes 


Supports, where needed, negotiations for people’s 
time 


a epalci program oversight reviews on a periodic 
asis 


Ensures process improvement activities progress in 
line with documented budgets and plans 


Performs or delegates review/approval of WG 
deliverables 


phic visible Support for the Process Group and 
S 
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Process Group 


@ Focal group for action planning, process improvement, 
technology insertion, etc. 


@ Channel for institutionalizing knowledge of process 
methods, practices, and technology 


@ Champion of change 
@ Facilitator of improvement efforts at all levels 


Sample Improvement 
Infrastructure 


& Organization 
© Project 
© Individual 


Process Improvement Infrastructure Development Organization 


Senior Management | 


Middle Management 
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What Does the Process Group 
Have to Know? 


@ Senior management @ Knowledge of software 
strategic direction Support activities such 


@ Organizational culture as QA and CM 
¢ Organizational structure Managing Change 
@ General knowledge of Team Building 


application domains @ Collaborative Consulting 
@ Knowledge of modern Skills 

software engineering 

techniques and 


methods @ Project Management 


@ Knowledge of software @petrics 
engineering standards 
(DoD, MOD, IEEE, ISO) 
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@ Technology Transition 


Purpose of the 
Process Group 


@ Coordinate up, down, and across the 
organization 


Conducts periodic SMAB reviews (at least quarterly) 


Conducts periodic Steering Committee reviews (at 
least monthly) 


Facilitates the activities of the Working Groups 
(daily) 


Promotes technical awareness and education about 
process improvement 
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Develop the action plan for a specific 
focus area 


Develop new or improved processes, 
procedures, guidelines, templates, 
etc. 


€ Based on a formally documented 
and approved plan 


Identify and evaluates technologies 
based on the organization’s and 
project’s processes 


Suggest and/or develop training plans 


Support the piloting of those 
processes 


Groups and project members 


Processes 
© 2006 Kasse Initiatives, LLC 


Version 2.2 


Working Groups (WGs) 


Sample Improvement 
Infrastructure 


Process Improvement Infrastructure Development Organization 


Senior Management | 


Middle Management 


Proj it 
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@ Evaluate pilot performance and revise the processes as necessary 
@ Report process improvement progress to the Steering Committee 
@ Share lessons learned on the Working Group with other Working 


@ Support the institutionalization of the tested and approved 
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Proposed Membership of WGs 


@WG Leader - (250%) 

Core Members - (230%) 

Process Group Representative - (£=100%)) 
@QA Representative - (230%) 

@ Members - (220%) 

@ Consultants - (5-10%) 

@Reviewers - (5%) 
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Middle Managers 
as 
Process Owners 
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Middle Managers as 
Process Owners 


Process owners are often associated with those 
individuals who have the most technical expertise 
to lead a working group 


@ Middle Managers are the “owners” of the resources 
that must make the project successful AND the 
process improvement effort to happen 


@ When Middle Managers are the Process Owners, 
Senior Management is ensured that his or her 
vision and business objectives are being 
considered for each piece of the process 
improvement effort 


Process improvement is no longer an exercise to satisfy the 
requirements of a model such as the CMMI 
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Middle Managers as 
Process Owners - 2 


@ Middle Managers take the responsibility to ensure that 
the Process Group and especially the Working Groups 
have the proper resources, tools, and guidance to 
develop new processes or review existing ones and get 
them deployed on the projects 


@ Middle Managers must communicate with the Process 
Group and the Working Group members that are 
performing the process improvement activities for their 
focus area to: 


& Understand what progress is being made against the CMMI and 
other related standards and models 


& What barriers are holding the process improvement progress 
back 


& What changes must be made to make the process improvement 
effort more effective 
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Middle Managers as 
Process Owners - 3 


@ Middle Managers can: 


Offer the Working Groups guidance based on their 
interaction with the Senior Management team 


© Effectively utilize their power and status to overcome 
problems hindering the process improvement 
progress 


Run interference for the change team 


Protect resources for critical projects including the 
process improvement project so that all necessary 
work gets done 


Continually update Senior Management on change 
process steps 
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Middle Managers as 
Process Owners - 4 


@ Middle Managers can then ask for measures from the 
processes that are being developed and deployed on 
the projects and put themselves in a position to make 
risk management decisions based on data from process 
measures and not just “hope” 


Increases their overall decision making capability 
Puts them into a “control” position 


@ Senior Managers can expect accurate process 
improvement progress reporting 
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Summary 


@ Make Middle Managers the Process Owners ensures 
that: 


& Ensure that Senior Management’ vision and business 
objectives are being considered for each piece of the process 
improvement effort 


& They take the responsibility to ensure that the Process Group and 
especially the Working Groups have the proper resources, tools, and 
guidance 


<& They communicate regularly with the Process Group and the Working 
Group members 


© Their power and status are effectively utilized to overcome 
problems hindering the process improvement progress 


They protect resources for critical projects including the process 
improvement project 


& Accurate process improvement progress is reported to senior 
management against vision and business objectives 
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Case Study 
ING (Postbank) 


Process improvement effort started in June 
1996 with training and an assessment 


@ Process Group leader was given responsibility 
for the success of the process improvement 
effort and for achieving Maturity Level 2 


@ Middle Managers were not involved and 
frequently put up barriers to the Process Group 


@ Quality Assurance was not treated seriously 
@ Measurement existed at a bare minimum level 
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Case Study 
ING (Postbank) - 2 


@ After one full year had passed there was little 
progress to report 


@ Middle Managers were identified and assigned 
as “Process Owners” 


Hated Process Consultants 
Hated CMM 
Hated Process Improvement 


@ Senior Manager held monthly process 
improvement progress meeting and asked 
Middle Managers what the progress was in 
regards to the focus area to which they had 
been assigned 
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Case Study 
ING (Postbank) - 3 


@ Quality Assurance became Quality 
Management 


Core Team Members had 15-20 years of experience 


as developers, managers, and supplier managers 
(1%) of total development staff of 600 


Every project had a less experienced Quality 
Coordinator 


@The cultural change component of the process 
improvement effort was recognized and one 
culture coach was trained and assigned to 
every 10 developers 
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Case Study 
ING (Postbank) - 4 


@ Chief Architect became head of organizational- 
level CCB representing the architecture and 
business direction of the organization 


Process Group gained adequate numbers of full 
and part-time (250%) 


@ Middle A enla ‘Se Started to receive quantitative 
information about the successes and failures of 
the process improvement initiatives they were 
Supporting putting them in the control position 
they had always sought 


@ Senior Manager clearly saw the process _ 
improvement initiative supporting their vision 
and business objectives 
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Thank You 
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© 2006 Kasse Initiatives, LLC Version 2.2 CMMI - Mid Mgrs — Process Owners - 46 


ral Kasse Initiatives 
Contact Information 


Pamelia S. Rost 
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Business Development 
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United States of America 
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Kasse Initiatives 
Contact Information 


Ingo Tegtmeier 
Operations Manager 


Kasse Initiatives LLC 
Niedereschacher Strasse 6 


78052 Villingen- 
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Germany 


+49 7721 407 851 — 
Business 
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Missile Systems 


You Shouldn’t Have 
To Be An Engineer 
To Understand Your 
Process! 
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Missile Systems 
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Missile Systems 


Have you ever noticed how dry process is? 


Process for the Sake of Process Helps No One! 
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Processes are written to reflect the ” fthee 
experience of the author Pee AYESEMN 


» Software Engineering led the process 
charge in the ’80s 
— Used Flowcharts 
— CASE tools 
— “data repositories” 

» Object-oriented influences in descriptions 
— Subroutine processes 
— Instantiations of process 

=» Threads of processes (function points) 

» Written for the worst case scenario 


» Processes written to stand alone not to 
integrate with each other 


Have you ever seen a GenXer use a flow chart? 
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R 
Missile Systems 


Evolution of Process 


=» Process is not just for 
software anymore 


=» More and more “softer skills” 
USE Process 
= They are picking up our 
“bad” habits 
— E.g. A parking instruction that is 
4 pages long 
— Directives that contradict training 
materials 


Who wants to read all this, anyway? 
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Missile Systems 


How do we see processes in real-life? 


=» One page fold out when you buy a 
computer 


=» Kneeboards for pilots 

=» Help bubbles onscreen 

=» The mysterious stranger that answers 
your questions in Final panies or Second 
Life > 

= Help files 

» FAQ'S 

» How your McDonalds order 
is completed 


Are we confusing process requirements with what we really need? 
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Many different types of people use our 
processes 


Missile Systems 


» Many Authors 
— Carl Jung 
e Myers-Briggs 
e Florence Littauer 
e True colors 


= Different types 


© TRUE COLORS 


— Sensing SEARCH INSIDE!" 
— Thinking 
— Intuiting 
— Feelin in 
_ Pet 
Tree 


We are not all the same; we all positively contribute to the cause 
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Missile Systems 


Processes need to be understood 


= Issues with divergent 
understanding of processes 
results in classic interface 
failures and defects. 


=» The more types of people 


eo 
that work together on the \@ ° 

team the more solutions they - 

will come up with. 

= Important to remember that 


Xer’s and Millenniums have 
a different perspective than 
Boomers. 


Can the new workforce use the process as it is written today? 
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Missile Systems 


Generational Diversity Who They Are 


a Traditionalists: 1900-1945 
— Oldest and second smallest generation in the workforce 
— Experienced WWI and WWII, as well as the Great Depression 
— Faithful to an industry as well as an institution 
— Many are approaching retirement and are taking part time jobs 
» Baby Boomers: 1946-1964 
— Largest generation in the work force 
— Raised in an era of extreme optimism, opportunity, and progress 
— Witnessed Vietnam and Watergate 
— Didn’t learn to use computers until after they had entered into the workforce 
— Given credit for inventing the 60 hour work week 
— Team oriented, but highly competitive due to their large numbers 
— Are results driven but desire personal gratification 
— Tend to enjoy office politics, Workplace relationships are very important 
— Faithful to an industry 


Does Generational Diversity Affect Process? 
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Generational Diversity Who They Are 


Missile Systems 


Gen X’ers: 1965-1980 
— Second largest generation in the workforce 
Latch-key kids 
Raised with TV's, microwaves, videocassette recorders, and personal computers 
Grew up in a time of rising crime, drug use, and divorce rates of 40% 
Need to know about the big picture, not just their piece 
— Technologically savvy and independent 
— Tend to dislike office politics, Seek a work/life balance 
— Every job is a stepping stone 
— Can be self-protective and skeptical 
Millennials: 1981-1999 
— Smallest generation in the workforce 
Many are still in school, but some of the older millennials are entering the workforce 
Never knew a time without computers 
Parents could arrange their schedules around their kids' lives 
Love technology and are extremely confident 
Able to synthesize information and interpret what is important to them 
— View adults (elders) in their lives as people who will help them solve problems and will 
work in their best interest 
How Can We Create Better Process for X’ers and Millennials 
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Missile Systems 


The Biggest Potential for Problems: X’ers and 
Boomers 


» Ways for X’ers to better work with Boomers 
— Show respect- be receptive to learning from the experience they have 


— Choose face-to-face conversations- technology came late to them and can be 
impersonal 


— Give them your full attention 


— Play the game- “Baby boomers are diplomatic and particularly adept at 
navigating politically charged environments” 


— Learn the corporate history- X’ers may come in with great ideas that require 
change, but understand where the company has come from first 
» Ways for Boomers to better work with X’ers 
— Get to the point- state objectives clearly 


— Use email- a well worded, concise message can be more convenient than a 
meeting 


— Give them space- X’ers are independent and can figure out the best way for 
themselves to get results 


— “Get over the notion of dues paying”- X’ers grew up with parents who worked 60 
hour weeks and had high divorce rates. They want to put in their 40 and go 
home. Rate them on ability, not time worked. 


— Lighten up- be open minded to new ideas and different ways of doing things. 
It’s okay if work is fun. 


Working Together We Can Improve Our Process 
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Learning Styles 


Auditory- Tend to prefer verbal instructions or 


talking about it with someone else. May 
become distracted by sounds or noises. Tends 
to forget faces but remember names or 
remember what you talked about. Prefers the 
telephone. Enjoys dialog and conversation. 


Kinesthetic & Tactile- Tend to gesture and 
use expressive movements. May become 
distracted by activity around them. They 
remember best what you did together. They 
prefer to jump right in and try it. Tend to ignore 
the directions and figure it out as they go along 


Visual- Tend to seek out pictures or diagrams. 
Look at directions and the picture. Tend to 
forget names but remember faces. Prefer 
direct, face-to-face, personal meetings. May 
become distracted by untidiness or movement 


Missile Systems 


Which of these styles are your process written for? 
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Good Teams Need Well Rounded Process “" °*""" 

» Robust teams include all types of 
learning styles 

= As the Boomers retire can your new 
workforce use the process as it Is 
written today? 

= What is the impact on the company 
when processes are too complex for 
the new user? 


=» Verbose process Is just wrong!! 


True integrate teams need process that fits multiple styles 
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Missile Systems 


Things to Consider Creating Writing Process 


=» Processes need to be understood by all learning styles 


» Processes are tools to help one do things better, not just 
literary volumes 


=» The more simple the process appears, the easier it is to 
follow 


= Be open to teaching the process in many different ways 


=» Put your self in the users spot Is it still 
the best process 


=» Go out of your own comfort zone to 
accommodate different styles. 


a» Have fun!! 


Process that are deployed in more than one style are more effective 
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Customer Success Is Our Mission 


Questions?? 


Thank you for your participation 


Raytheon 
Presentation Outline Missile Systems 


I Introduction 
IL. How are processes captured 
a) SW wrote 
b) Flow Charts, Spaghetti, CASE Tools, Context Diagrams 
ne Evolution of models 
a) Now looking at all types of disciplines/industries 
b) Don’t understand well structured not well integrated 
IV. Excerpt of IPO vs. Excerpt Securtiy/HR 
V. Personality Types 
a) Process looks at 2 of 4 types 
»» Analytical and Action 
0) Blue and yellow have no idea what they are talking about 
¢« Good team is well rounded all types of 
Vi. Take Aways 
VIL. Questions 
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| Have Had My CMMI Appraisal... What 
Do | Do Now? How to Establish a 
Process Improvement WBS 

Aldo Dagnino 
TO KNOW. Director of Manufacturing Solutions 


THE 
POWER 


Introduction 


Work Breakdown Structure Characteristics 
Work Packages 


A Practical Example 


Problem Statement 


= Creating a work breakdown structure (WBS) 


that will serve as a basis to create a process 
improvement plan utilizing the Final Findings 
obtained after a CMMI Appraisal 


Tit 


Assumptions 


= A CMMI Appraisal has been recently conducted 
focusing on specific CMMI process areas (PAs) 


= Findings from the appraisal provide details on 
strengths and weaknesses on the PAs in scope 


= Details on weaknesses are actionable 


= The Process Improvement team has experience 
developing plans 


CMMI Project Planning Process Area 


= SG 1-— Establish Estimates 


¢ SP 1.1 - Estimate the Scope of the Project 


— Establish a top-level Work Breakdown Structure 
(WBS) to estimate the scope of the project 


e SP 1.2 — Establish Estimates of Work Product and 
Task Attributes 


— Establish and maintain estimates of the attributes of 
the work products and tasks 


¢ SP 1.3 — Define Project Lifecycle 


—- Define the project life-cycle phases on which to scope the 
panning effort 


e SP 1.4 — Determine Estimates of Effort and Cost 


— Estimate the project effort and cost for the work 
products and tasks based on estimation rationale 


CMMI Project Planning Process Area 


= SG 1-— Establish Estimates 


¢ SP 1.1 - Estimate the Scope of the Project 


- Establish a top-level WBS to estimate the scope of the 
project 
e SP 1.2 — Establish Estimates of Work Product and 
Task Attributes 


— Establish and maintain estimates of the attributes of 
the work products and tasks 


¢ SP 1.3 — Define Project Lifecycle 


— Define the project life-cycle phases on which to scope the 
panning effort 


e SP 1.4 — Determine Estimates of Effort and Cost 


— Estimate the project effort and cost for the work 
products and tasks based on estimation rationale 


Sub-practices _ 


= Develop a Work Breakdown Structure 


= Identify work packages to specify tasks, estimates, 
responsibilities, and schedule 


The Work Breakdown Structure — 1 — =e 


= Serves to identify the work components needed to 
complete the development project 


= \|tis the basis for size estimates 


= The WBS Is structured according to the way in which the 
work will be performed in the project 


= The WBS reflects the way the project will be planned and 
tracked 


= The WBS organizes the work into a hierarchical set of work 
relationships needed to accomplish the objectives of a 
project 


The Work Breakdown Structure — 2 — 


_ Coes | 


A top-level WBS serves to structure the initial estimates 


The WBS divides the overall project into an interconnected 
set of manageable components 


The WBS provides a scheme for identifying and organizing 
the logical units of work to be managed, which are called 
“work packages” 


The WBS provides a mechanism for assigning effort, 
duration, schedule, and responsibility 


The WBS is used as the underlying framework to plan, 
organize, and control the work done on the project 


The WBS allows to hierarchically organize the project 


The WBS provides a framework to track progress in the 
project 


The Work Breakdown Structure-3- 9 


= The WBS allows to identify: 
° Tasks associated with deliverables and supporting 
activities 
e Tasks for skill and knowledge acquisition 
¢ Tasks for development 
¢ Tasks for integration 
¢ Tasks for management of the project 
e Risks and associated mitigation strategies 


The WBS identifies at the lowest level of the tree 
hierarchy “work packages” (WPs) 


WPs are units of work that can be assigned to a sub- 
team and can be individually tracked 


WPs is assigned a unique ID to be tracked in the 
Process Improvement Project Plan 


= The WBS Contains, at Its lowest level, information 
related to: 


¢ The scope of the work to be performed 
¢ A list of deliverables associated with the work packages 
e A list of required resources to complete the work 


¢ Required skills and knowledge to complete the work 


S¥ste ae 


Six-level WBS Tree Hierarchy 


aa Zz T, Kasse (2004) “ A Detailed Look 


at the Work Breakdown Structure 
(WBS) and Work Packages”, 

2004 SEPG Conference in Orlando, 
Florida 
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Work Package Template 


Draiact Nama 


An Example: Geographically Dispersed SW Development 
at SAS 


= Sister development organization in India 
= Often remote collaborative software development 


= Management of remote development often based 
on personal relations 


= Certain projects require more structured approach 
to manage remote software development, 
especially for newly formed groups 


Findings CMMI Internal Class B Appraisal -1- 


» Strengths 


Organizational policy to manage external suppliers 
exist 


Supplier Agreements for COTS products are developed 
COTS products are evaluated against requirements 


Supply Chain Management handles the purchasing of 
commercial components for HW, SW and contractors 


Remote sister Development organization uses same 
RE, CM, and defect tracking tools as Cary team 


QSAS | Fes. 


Final Findings 


» Weaknesses 


¢ No organizational policy to manage remote 
development with sister organizations 


¢ No organizational procedure to manage 
development activities with sister organizations 


No formal agreements are established with 
remote development teams in sister organizations 


Transition of work products provided by sister 
organizations performed in informal manner 


Final Findings —3- 


» Recommendations 


¢ Develop SAM Procedure and Process Map for remote 
development 


¢ Pilot SAM procedure 


¢ Integrate R&D SAM and Supply Chain Management 
procedures 


e Institutionalize SAM Process 


Project Goal Level WBS 
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Project Level WBS 


S¥ste ae 


Task Level WBS 


Sub-task Level WBS 


Project Goal 
Level 


Project Level 


Task Level 


Level 4 


Develop SAM R pects AM 
Sub-task Level Procedure and Brocediire 
Process Map ID#1.1.1.2 


ID#1.1.1.1 
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Work Package Level WBS 


S¥sts See 


Effort Level WBS 


Work Package and Effort 


Project Name 


Develop and Implement Supplier Management 
Process for Remote Development Activities 


Revision # and Date 
V 3.1 June 12, 2006 


Work Package Name 


Work Package # 


Brainstorm R&D SAM Procedure 1.1.1.1.2 


Work Package Description 


A group of selected domain experts in managing remote development in the past will brainstorm 
a viable process to follow in order to better manage remote development activities. An initial 
process map document will be created as the brainstorming session Is carried out. Minutes of the 
meeting will be taken. The brainstorming session will begin with a presentation of the CMMI SAM 
PA by the SAM EPG 


Deliverable(s) ; 
Deliverable includes a first draft of the SAM process map and minutes of the brainstorming 
meeting that will be used as input to WP # 1.1.1.1.3 


Guidelines for Work 
Follow the organizational standards required in the development of new processes in the 
organization and process maps 


People Skills Size Estimate Time Estimate 
SAM EPG Lead oui ey one 
facilitation expertise 2 page 
Project Manager 1 Project Mgmt expertise process map 8 hours per 
Project Manager 2 Project Mgmt expertise Hr person 
; evel o 
Development Manager in India Development expertise difficulty 


Development Manager in Cary Development expertise 
Scribe Mapping expertise 


WBS in Detail Created by SAM EPG 


Develop and implement SAM process (policy, procedure, and process map) for remote 


development 
¢ Develop SAM Procedure and Process Map for remote development 
- Design and Develop SAM Procedure and Process Map 
— Review Supply Chain Management Policy and Procedure 
— Brainstorm among SAM team new R&D SAM procedure 
— Document SAM process map 
— Document SAM procedure 
- Develop few relevant SAM metrics 
— Metrics required within the SAM procedure 
— Metrics to measure effectiveness of SAM procedure 
- Document required SAM templates 
— Remote Development Agreement 
— Metrics template 
e Pilot SAM procedure 


- Identify two pilot projects 

- Implement pilot projects identified 

- Implement pilot SAM metrics 

- Monitor pilot projects 

- Document lessons learned 

- Refine and finalize SAM procedure 
e Integrate R&D SAM and Supply Chain Management procedures 

- Identify all of the supply chain processes 

- Review R&D SAM and SCM processes jointly. 

- Identify interface points 

- Ensure that SCM representatives participate on the contract management team 
¢  Institutionalize SAM Process 

- Develop SAM Policy 

- Develop SAM Process training materials 

- Train organization relevant stakeholders 

- Implement new SAM process in all new development projects 

- Monitor SAM process in all projects 


Conclusions 


» It is necessary to go deeper than the information 
typically obtained in the Appraisal Final Findings 


= Carry the work breakdown structure tree to the level at 
which you can define self-contained and measurable 
tasks that can be assigned to specific individuals or 
smaller team 


= Decide on how you want to track your Process 
Improvement project 


= Make sure you involve the people that are and will be 
directly involved in the specific process being 
improved 
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_ Software Design 
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no. November 15, 2006 
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Richard L. W. Welch, PhD 
Chief Statistician 

April King 

Systems Engineer 
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Topics 


= Organizational overview 
= The challenge of SPC early in the life cycle 


= Practical difficulties & how we addressed them 
= Process selection 
= Data rates 
= Statistical innumeracy 


= Ongoing issues 


= Math details (for reference only) 
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Corporate Overview 


= Comprehensive business 
portfolio to address the 
defense and government 
markets 
— Systems integration 
— Military aircraft 
: — Unmanned aerial vehicles 
— C4ISR 
— Defense electronics 
| — Information technology and 
networks 
— Naval shipbuilding 
| — Space and missile defense 
| 


= 2005 sales of $30.7 billion 
= 125,000 people; 50 states; 
25 international countries 


= Headquartered in Los 
Angeles, CA 
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gu esa 
77% 


IMM AR 


Ef as las Fae 


Northrop Grumman Integrated Systems 


E-2C/AHE 


Re NATO Ags  ADYVANS=0 2 


™’ ¢ PTS. 
~ 
a» 


EA-186 ICAP III 


Premier Aerospace and Defense Systems 
Integration Enterprise 
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Ww AGS&BMS Major Sites 


/ 


Level 5 CMMI-SE/SW/IPPD/SS 6/06 
Level 5 CMMI-SE/SW 4/05 
ISO 9001:2000/TickIT 9/04 


Pp Lf O/C ig (a e/a E774677" 1 
5 a 
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Where We Were 


= First demonstration of CMMIS™ Level 4 and 5 
capabilities focused on code inspections and peer 
review of test plans, procedures and reports 


= High data rates inherent in these “back end” processes 
helped us to understand and overcome statistical 
difficulties 

= We gained practical lessons learned on the obstacles 
that had to be overcome 

= Senior author developed innovative log-cost model 
presented at 2005 CMMIS™ Technology Conference and 
User Group 


= Positive business benefit (& successful CMMIS™ Level 
5 appraisal) resulted in strong senior management 
support to expand 
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SPC Determines the Voice of the Process 


Quantitative Project Management 


ASU Log Cost Model 
Using Lognormal Probability Density Function 


A stable process 
"Average performance _ operates within the 
. control limits 99.7% 
of the time 


LN (Hours per New/ Mod LOC) 


@ ee © RS RS 
ss ss BS Ps oS O 
Se ee a es 
Review Closed Date 


Analysis of 
= Special cause variation focuses on recognizing & 
preventing deviations from this pattern 
= Common cause variation focuses on improving the 
average and tightening the control limits 
= Offering opportunities for systematic process improvement that 
NGC & industry benchmarks indicate will yield an ROI 


| averaging between 4:1 & 6:1 
NORTHROP GRUMMAN 
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Challenge 


= Desire to introduce successful SPC techniques for 
quantitative project management in the “front end” 
system and software design phases 
= When coding starts 
= Product development is one-half over 


= Opportunities to recognize and correct special & 
common cause variation in the design process are gone 


DoD studies indicate first-year decisions 
determine up to 70% of total life cycle 
cost. Early, effective statistical control 

offers great practical benefit 
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Practical Difficulties at Level 4 


= Getting started = Cautionary note: this 
= Selecting good assumes you have also taken 
candidates for statistical care of the basics (CMMI°™ 
management Level 3) 
= Little historical data & " Buaget and charter 
inherently low data rates = Project impacts 
= Process model for = Metrics infrastructure across 
successful statistical engineering 
control = Metric definitions 
= Statistical innumeracy = Data collection 
= Discipline needs to own mechanisms 
the right skill set = Consistency of processes 


across projects 
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Getting Started 
Process Selection for Statistical Management 


= Statistical control is imposed on sub-processes at an elemental 
level in the process architecture 
= Processes are selected based on their 
= Business significance — “sufficient conditions” 
= Statistical suitability — “necessary conditions” 
= Business checklist 


= Is the candidate sub-process a component of a project’s defined 
“key” process? 
= Is it significant to success of a business plan goal? 
= Is it a significant contributor to an important estimating metric in 
the discipline? 
= Is there an identified business need for predictable performance as 
projects execute the subprocess? 
= Cost, schedule or quality 
NORTHROP GRUMMAN 
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Statistical Checklist 


PRINCIPAL FACTORS INVOLVED IN SUB-PROCESS SELECTION FOR STATISTICAL 
PROCESS CONTROL 
PRIMARY QUESTION SUPPORTING QUESTION DEMONSTRATED 
INDICATOR 


Are data collected? Does the data collection system Data collection system 
require update or redeployment? ready for deployment 


What is the data rate? That is, how 
often will the process be repeated on 
the project? 


Are there historical performance 
data? 


Has a control metric been identified? 


Does a baseline exist for the control 
metric? 


Will the process be repeated frequently 
enough to develop control limits if such 
limits do not exist from baseline 
historical performance? 


Stable performance: Will the process 
be performed in roughly the same 
manner as on previous projects? 


Has Statistical analysis of past project 
performance identified measures that 
are indicative of overall process 
performance? 


What are the average and statistical 
variation of previous performance? 


At least 20-30 data points 
exist or will be produced 


A documented procedure or 
training materials are used 
by those performing the 
process 


Control metric can be 
computed from the 
collected data 


Performance excursions 
outside the contro! limits 
can be identified & 
attributed to their root 
causes 


Do specification limits exist for Do limits exist beyond which process Specification limits are 
process performance? (Optional) performance is deemed unacceptable? documented 


gl 
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The High Maturity Data Dilemma 


Unstable performance 


Stable performance 
Improved performance 


At, : At, & At, : 

¢ Process selection ¢ Identify improvement proposals 

e Analysis of suitability ¢ Evaluate & prioritize proposals 
for SPC ¢ Select improvement 


e Pilot improvement 
¢ Deploy improvement 


We can minimize At, At, & At, by careful 
management, but the length of the data runs will 


depend on the periodicity of the process itself 
AGS&BMS-PR-06-122 
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Reality at the Front End of the Life Cycle 


= 9 candidate systems & software design processes 
identified for statistical management 


= Small sample issues present 

= Control charts may look OK 

= Further analyses show data skews 
= How do we find the best model? 


Physical Model Peer Reviews Physical Model Data 


13 


Review Effort (Per Change) 
\ fi fl 1 i 1 
Bo ee 
a 
oe 
Sy ia 
b 
E 
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Q 


fs) 
Observation 


Anderson-Darling Test for 7 —:s| 


Normality p < 0.025 
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Finding the Right Model Is 
Not a Random Hunt! 


= Minitab’s Individual Distribution Identification tool 
= 14 simultaneous goodness-of-fit tests 


Probability Plots for Physical Mode! Data 
Cones ot 


Weibull - 95% Cl 


2-Parameter Exponential - 95% Cl 


=" 7 models “pass” - 
what next? 


=" What happens 


i ae when you collect 
He ; one or two more 
data points? 


= In this case, one 
of the models is 
actually correct! 
= But you'll never 
be able to prove 
it like this 


Percent Per 

R y 2s 8 B y 2g 8 
m8 ty 8.8 m8 ty 8.8 
2 y 2 
2 gg 
Ey a 
3 Sih: mhKe z 
3 e 
gs ca 
oo 
g 

Percent 
m 8 § 8 
3 
= 


Probability Plots for Physi 


Largest Extreme Value - 95% Cl 


1 2 
1 : F 
hange) Review Effort (Per Change) Review Effort (Per Change) Review Effort (Per Change) 
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More Data 


«*) "Larger samples don’t 
solve everything 
= How can you show 
this process is 


Proposed Spec Change (PSC) Peer Reviews 


1 
i 
UCL 


Stable and in- 4 

control? Boy hel h 
= And if we can't que Dai 

control the effort, cre reel 

how can we —— 

compare defects In Anderson-Darling Test p < 0.005 


a meaningful way? 


NORTHROP GRUMMAN 
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Stabilizing the Data 


= Senior author’s presentation ears 
at last year’s CMMIS™ eee\» Logarithms 
Technology Conference a 3, 
demonstrated the applicability [FF ae 

of a log-cost model to control | 


software code inspections 


= Peer review effort behaves like commodity prices in the 
short term 

= The math is applicable to any per unit “price” of a 
repeatable engineering effort 


= Using natural log transforms data to normal distribution 


NORTHROP GRUMMAN 
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Log Transformed Series 


Physical Model and PSC Examples 


Physical Model Peer Review 
Electronic Meetings/Engineering Checks 


T T T T T T T T T 
8626 8666 8689 8729 8711 8723 8769 8879 8872 
Review 


Worksheet: Physical Model. MTW; 5/3/2006 


7 a 
5 
é | 
i 
% LCL 


Proposed Specification Change 
Electronic Meeting - Engineering Check 
7472 9155 9759 


Training ‘Expanded 


Log Review Effort (Per Change) 
6 rl 
Q 


ne T T T T T T T T T T 
7472 7571 8536 9004 9023 9027 9120 9298 9326 9300 9822 
Peer Review Number 


Process Improvement Initiated - Stage 1 & 2 Traini 
Worksheet: PSC.MTW; 6/28/2006 


Anderson-Darling Test p < 0.329 


= Can you eyeball the difference 
with slide 13? 
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Anderson-Darling Test p < 0.919 


= Did you realize slide 15 had 
= 2 buried improvement cycles? 


"Only 1 special cause point - which 
wasn’t flagged? 
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The Two Models Compared 


Raw Data 


- = 11% false alarm rate 
(Chebyshev’s inequality) 
= Penalizes due diligence in 
peer reviews 
=" No meaningful lower control 
limit 
" Does not flag superficial 
reviews 
= Arithmetic mean distorts the 
central tendency 


= Apparent costs can exceed 
budget 
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Log-Cost Transformation 


False alarms minimized 
Meaningful lower control limit 
Geometric mean preserves 
the budget 
" OK, you still have to find the 
antilog 


Dramatically improves the 
evidence of stability and 
control 
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Overcoming Statistical | nnumerac 


Success Factors 


= Minitab 
= “Dark green belt” training 


= Curriculum tailored to focus on applied statistical 
techniques and Minitab familiarity 


= Deming principle applied in the class room 
= In God we trust, a// others bring data 
= Lean and process management training covered in 
other courses 
= Green belt community of practice 


= Chief statistician 


Key success factor: management 
recognition & support for the investment 


NORTHROP GRUMMAN 
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Statistically Managed Processes 


Systems & Software Baselines 


= System Engineering = Software Engineering 
= System design & system = Software design peer reviews of 
hi architecture peer reviews of = Software threads 
= System threads = Physical model 
= System model (structure = Component/task descriptions 
diagrams) 


= Data model 
= Software code inspections 
= Software build process 
Software build returns 
= Software test returns 


= Physical model 
= UML diagrams 


=» System & software 
requirements peer reviews of 


= Proposed specification 
changes (PSCs) 


Most are applicable to early life cycle phases 


Note: baselines highlighted in blue use the log-cost model. NORTHROP GRUPRIAN 
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Statistically Managed Processes 


Other Engineering Baselines 


_. #; = Test & Engineering = Vehicle Engineering 

= Peer reviews of test plans, = Electro-mechanical drawing 
procedures & reports errors 

= System Integration Lab (SIL) = Vehicle subsystems (i.e., crew & 
scheduling equipment) drawing errors 

= Avionics = Logistics 

= Discrepancy Inspection =" AF Tech Order (AFTO) 
Report (DIR) processing processing of the 

» Avionics Drawing Sign-off = Total contractor schedule 

= Field Service Engineering = LSA group schedule 
Request (FSER) processing = Integrated electronic technical 

= Management of seller issues manual (IETM) delivered quality 


Statistically managed baselines span all 
life cycle phases & engineering disciplines 


NORTHROP GRUMMAN 
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Ongoing Steps 


As a consequence of the successful expansion across 
all engineering disciplines and all life cycle phases, we 
have recognized the need to grow the infrastructure of 
trained, experienced statistical practitioners at the mid- 
level 


= Sector standards for certifying Green Belts, Black 
Belts & Master Black Belts 


= Training 

= Project portfolio 

= Developing Black Belts cadre 
= Future Master Black Belt cadre 
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Pitfalls 


= Dealing with what was taught, but not learned in 
Green Belt class 


= Monthly data won’t work 
= It takes 20-30 months to accumulate 20-30 points 


" After-the-fact hypothesis testing is not a designed 
experiment 


A time series is not a random variable 
“Between the lines” doesn’t mean in-control 


SIL Utilization Planning Performance Build Times by Build Date 
Z TSSR - SW Development | Lab Last 50 Builds are displayed. 
: J 
oa J 
> g ; 
q | © 0 00 0 © 0 © © 0 © © © © © © © 0 © © ¢ © 0 0 © reall < 
5 2 a 1 
Ue Pay : 
< | 3 
2] Z \ \ 
: | LCL el - 

Mh OSyy, i Oy, Ty Sou, iy 22 lib, Cy Creo Aa. © Ld 
xu ‘Ar op Me, Otto] engin Cal un.ofl nce lut as Aug. Ota -op SHG 58, aoe oe | 
Date 
Positive = Worked more Shifts than Planned LcL 


Negative = Worked less Shifts than Planned T T T T 
Worksheet: Worksheet 1; 10/9/2006 Worksheet: Worksheet 1; 10/9/2006 
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123 


Richard L. W. Welch, PhD 
Northrop Grumman Corporation 
(321) 951-5072 
Rick.Welch@ngc.com 


24 


AGS&BMS-PR-06-122 


April King 
Northrop Grumman Corporation 
(321) 951-6057 
Al.King@ngc.com 
NORTHROP GRUMMAN 


NORTHROP GRUMMAN 


DEFINING THE FUTURE 


Reference 
Slides 


TAO) iil erctsjim (ake lasy pm pessyo)p)tsitio)p) 


Math Details 


= Consider a stochastic process... ., X_5, X_1; Xo; X1 Xo, 
... that represents an asset price recorded over 
time, like a daily sequence of prices for shares ofa 
stock or other commodity 


= We assume at time f that the realization x, of X, is 
known, but the realization x,,, of X,,., is unknown 


= The single-period log-return, /n(X,,.,/x,), is random 
and assumed to be normally distributed, at the 
given time tf 


= Under these assumptions, X,,.,/x, is a lognormally 
distributed random variable, and therefore, so is X,,, 


Math Details extracted from: 


http://www.riskglossary.com/articles/lognormal_distribution.htm NORTHROP GRUMMAN 
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Salient Properties of the Model 


= When log-returns are normally distributed, the 
corresponding prices are lognormally distributed 


= This model “is one of the most ubiquitous models in 
finance” 


= The distribution of log-returns and share prices have 
been validated empirically by many market studies 
accessible on the web 


= For short time periods in a stable market, the mean 
return is 0 


Quotation from: 


http://www.riskglossary.com/articles/lognormal_distribution.htm NORTHROP GRUMMAN 
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Lognormal Density Function 


X ~A[u,o7] Y =In(X)~ NU, 0°] 
E(X)=exp(u+o°/2) 
Var(X)= (exp(o”) — Lexp(2 ito’) 


Math details can be found in any standard mathematical statistics reference, 
see for example, http://en.wikipedia.org/wiki/Lognormal_ distribution. fae ala ace 
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Background 


=" An underlying principle of the CMMI model is the concept of 
a") conscious choices about how each process area will be 
implemented 


= For example, in implementing Configuration Management, the 
project planner must decide: 
#" Which work products to place under configuration control 
(e.g., deliverables, designated internal work products, acquired 
products, tools) 
= What levels of configuration control (e.g., author, informal, formal) 
are appropriate for each work product 


eat 


= This presentation will study the strategic implications of the 
CMMI on planning and implementing project processes 
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Topics 


= The Concept of Conscious Choices 


«= ©) = Key Choices in CMMI 
y = Configuration management 
; = Process and products audits 
= Verification, peer review, and validation 
= Others 


= Implications 
= Project planning 
= CMMI appraisals 
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The Concept of Conscious Choices 


= The CMMI was written to apply toa 
variety of project environments 


= Defense, commercial 
= Development, maintenance, services 
= Small to large project teams 


= The CMMI authors wrote the practices, expecting that different 
choices would be made concerning how they were applied 


=" E.g., “adequate”, “as appropriate”, “as needed”, “selected”, “identify” 


= You must the interpret goals and practices in light of your 
organization’s business objectives 


= BUT, you need to understand the intent behind the practice 
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Conscious Choices in CMMI Process Areas 


Level Focus Process Areas 


Continuous 
process 


Causal Analysis and Resolution 


ee) TUTE) Organizational Innovation and Deployment 


improvement 


4 Quantitatively| Quantitative Quantitative Project Management Levels 4 and rs) 
Managed management Organizational Process Performance have similar 


choices 
Organizational Process Focus 

Organizational Process Definition 

Organizational Training 

Integrated Project Management (for IPPD) 

Risk Management 

Decision Analysis and Resolution 

Requirements Development 


Technical Solution Verification and 
Product Integrati 


Verification Validation require 
Validation 


strategies 
Requirements Management 


Basic Project Planning 
project Project Monitoring and Control 
management Supplier Agreement Management 


Measurement and Analysis 


Process and Product Quality Assurance 
5 


Process 


3 Defined standardization 


2 Managed The Support 


process areas 
have several 
implied choices 


Configuration Management 
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GP 1.1 


GP 2.1 
GP 2.2 
GP 2.3 
GP 2.4 
GP 2.5 
GP 2.6 
GP 2.7 
GP 2.8 
GP 2.9 


GP 2.10 


GP 3.1 
GP 3.2 


GP 4.1 
GP 4.2 


GP 5.1 
GP 5.2 


Conscious Choices in CMMI Generic Practices 


Perform Base Practices 


Establish an Organizational Policy 

Plan the Process 

Provide Resources 

Assign Responsibility 

Train People 

Manage Configurations 

Identify and Involve Relevant Stakeholders 
Monitor and Control the Process 

Objectively Evaluate Adherence 

Review Status with Higher Level Managemen 


Choices 


Establish a Defined Process 
Collect Improvement Information 


Establish Quantitative Objectives foythe’Process 
Stabilize Subprocess Performance 


Ensure Continuous Process Improyvyement 
Correct Root Causes of Problems 
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Configuration Management 


SP 1.1 Identify Configuration Items 
Identify the configuration items, 


components, and related work products 
that will be placed under configuration 
management. 


= Must identify ALL the products that will be 
controlled — not just CDRLs or delivered 
products 

= Possible levels of control 


= Control board authorizes and controls 
changes 


= Designated authority controls 
changes 


= Author controls changes 
=» Anyone can make changes 


Any artifact 


produced bya 
process. 


Examples of work products that 
may be placed under CM include 
the following: 

e Plans 

¢ Process descriptions 

e Requirements 

¢ Design data 

e¢ Drawings 

e Product specifications 

¢« Code 

¢ Compilers 

¢ Product data files 

¢ Technical publications 
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GP 2.6 Manage Configurations 


Must decide which work products Any work products: 
are controlled and which are not Specs, designs, code, test documents, CDRLs, 


plans, reports, minutes, action items, etc. 


P 2.6 Manage Configdrations 


eee 
ulfder appropriate levels oreontrol. 


Must decide the 
appropriate level of control 
for each work product: 
Control board, management 
authority, author 
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Sizing the Amount of CM/ DM Needed 


= In the program management strategy, the project must decide 
—" which products to control, and what level of control 


» = Trade-off between the cost of more 
gt formal control and the risk of 
unauthorized changes 


= Must understand where the 
CM leverage points are 


=" Where does an unauthorized change cause significant problems? 


= Must understand the CM environment 
=" How much change is anticipated, and at what times? 
= Do the team members need more/less control? 


NORTHROP GRUMMAN 


Copyright 2005 Northrop Grumman Corporation 


Process and Product Quality Assurance 


= Must select the processes, work 
products, and services to be 
evaluated (audited) 


A SP 1.1 Objectively Evaluate Processes 
Objectively evaluate the designated 
performed processes against the 
applicable process descriptions, 
standards, and procedures. 


= Must decide what quality 
standards will be defined 
(e.g., coding standards) 
= Organization sets standards 
based on market needs and 
“brand image” 
= Project sets standards based 
on customer needs 


Objectively Evaluate Work Products and 
Services 

Objectively evaluate the designated work 
products and services against the 
applicable process descriptions, 
standards, and procedures. 


= Project process descriptions 
describe the process to be 
followed 


= Procedures are detailed 
process descriptions 


= Not all work products will have 
standards (e.g., coding 
standards) 


NORTHROP GRUMMAN 
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Generic Practice for Quality Assurance 


Could be interpreted to include 
both work products and processes 


GP 2.9 Objectively Evaluate Adherep 


Objectively evaluate adherence of th ainst its 
process description, standards, and procedures;> 


and address noncompliance. 


Must define “minimal acceptable” quality 
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Sizing the Amount of QA Needed 


= In the program management strategy, the project must decide 
<7 which work products and processes to define standards for, and 
how often to audit them 


= Trade-off between the cost of more quality and the risk of re-work 


= Must understand where the 
QA leverage points are 


=» Where does poor quality cause significant problems? 8. 


= Must understand the QA environment 
=" How much quality is expected, and of what type? 
= Do the team members need more/less auditing? 
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Verification 
Ensure selected work products meet their specified requirements 


SG1__‘— Prepare for Verification = “Prepare” goal suggests a 


Preparation for verification is strategy (conscious choice) 
conducted. = Integration, Verification, 
Validation 
User needs 


used for each: System requirements 


=" Which work products offer 
the greatest leverage? 


= Impact of errors, visibility 


Software requirements 


Code ft ere 
= How should the work product testing Software design 
be verified? Code 
= Peer review vs. testing vs. Soft F iietie 
inspection ortware code review 
= Efficiency, effectiveness gy 


System 


NORTHROP GRUMMAN 
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Validation 


Demonstrate that a product or product component fulfills tts intended 
use when placed in its intended environment 


SG1__s*~PPrepare for Validation 


a | Preparation for validation Is 
‘aes i conducted. 


Requirements 
review 
SP 1.1 Select Products for Validation 
oducts and product componen p 
to be validated and the validation methods~ 
that will be used for each. 


User needs 


System requirements 


= Which work products offer 
the greatest leverage? Software requirements 
= Impact of issues, visibility 


= How should the work product be Software design 
validated? 


E.g., user surveys, prototypes, 
milestone reviews, demonstrations, Software code 
formal testing gy 


System 
Operational Wor7HROP GRUMMAN 
testing 
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Other Conscious Choices 


Which decisions should be subject to 

a formal decision process (DAR) 

Integration sequence (PI) Cost, incremental confidence, 
testing insight 


What training to provide - 
organizational and project (GP 2.5) 


| What improvement information to 
_ | collect (GP 3.2) 
Which process elements to Cost, insight business value, 
quantitatively control (SPM, GP 4.2) ability to stabilize processes 


NORTHROP GRUMMAN 
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Implications 


*Z = In planning, the project manager must consciously make these 


a, choices 
A = Project defined process (tailored from the organizational standard 
a: process) identifies the processes to be used 


= Project plans identifies the choices made on how to implement 
these process 


= For example, the organization can develop planning templates 
involving fill-in tables to capture the choices 


= E.g., a stakeholder table (who, how, how often, etc.) 


= These choices must be clearly explained to appraisers 


NORTHROP GRUMMAN 
16 
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Summary 


*Z = Anunderlying principle of the CMMI model is the concept of 
— conscious choices about how each process area will be 
% implemented 


f = Recognizing the choices helps properly adapt the CMMI practices 
- to fit the context of the organization and its projects 


NORTHROP GRUMMAN 
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Background 


= Patterns are acommon way of expressing common contexts and 
aa problem-solution pairs, and are often used in software design 


* % = In CMMI-based improvement, patterns can be used to show how 
, | sets of improvement practices can be combined to achieve 
successful improvements 


= This presentation will introduce some typical patterns in process 
improvement 


NORTHROP GRUMMAN 
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The I DEAL Models™ 


IDEAL provides a usable, Learning 
understandable approach to Ve 5 7 
continuous improvement b * Document 
ae P y Revere md anatyze 
outlining the steps necessary to ‘organizational #35 ORe Acti 
establish a successful approach | cing 
i Dein 
improvement program bathe . 
Shad freeas ee 
Shanks te Set oo nlesct : ‘7 
improvernerd and establish mpeowgrmant deakicheg 
spororhip infrastructure pilots 
Pian, 
Initiating Appr aie a erect, 
currant installation 
pl tice 
Establish process 
Devel daar ah 
Fai kc ae on taaine 
ard doce amet ptioritiee Pan actions. 
: Set Tite PTE eS 
phase rasubs te faa } 


Diagnosing 


“IDEAL: A User's Guide for Software Process Improvement ,” Robert 


; NORTHROP GRUMMAN 
McFeeley, Software Engineering Institute, CMU/SEI-96-HB-001 
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Patterns 


NYeiavrtas 
Product Lines 


Product 


Process 


Organization 


oe 
informs 


= Patterns are acommon way of expressing common contexts and 
problem-solution pairs 


= The context is the organizational situation 
= The problem is what part of a effort needs to be accomplished 


= The solution is the grouping of practice areas and the relations 
among them that together address the problem for that context 


| | 
| Establish 


Establish | Operate 
Context | Production | Product Line 
| Capability 
m | £ + A : t | 71 
| | | 


What to _1_, Product le product Builder 


Rater ieee —_—- -—- — — -_— 
| 


fej ott 
| | 
Cold Start | In Motion | 


Adaptian Factory Pattern 


= The book Software Product Lines: Practices and Patterns defines 
12 patterns and 11 variants 


The Adoption 
Factory pattern is a 
composite pattern 
that describes the 
entire software 
product line 
organization. 
NORTHROP GRUMMAN 


Copyright 2005 Northrop Grumman Corporation 


Example - What to Build Pattern (Software 
Product Lines) 


Understanding Market Analysis Technology 
Relevant 


~~" Forecasting 
Domains a 
Product| | Market _" Technology | Technology 
Ser Pa 


Climate - Predictions | Predictions 
Domain ae GN Market 
Models Climate 


Justiicaton " 


i ery Building 
Product Set Business Case 
sar ies Business 
Line | 
Scope Case 
' 


Dynamic Structure 


NORTHROP GRUMMAN 
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What Benefits Would Patterns Provide? 


*Z = Address recurring CMMI adoption problems that arise in specific 
—" situations and present solutions to them 


= Document existing, well-proven CMMI adoption experience 


= Identify and specify abstractions that are broader in scope that 
single practice areas 


= Provide a common vocabulary and understanding for CMMI 
adoption 


=" Document, explain, and plan CMMI adoption efforts 
= Help manage the complexity inherent in CMMI adoption 


= Can be combined to build complex CMMI adoption solutions 


Adapted from Pattern Oriented Architectures: A System of Patterns, 
Bushmann, et al, 1996 and Software Product Lines: Practices and NORTHROP GRUMMAN 
Patterns, Clements and Northrop, 2005 
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Structure of a Pattern 


= Context: Organizational situation 
—/ = Defense contractor 

*. = Commercial contractor 
y = Small organization 


= Problem: What part of a effort needs to be accomplished 
= Adopt CMMI for the first time 
= Transitioning from CMMI v1.1 to v1.2 
= Transitioning from Level 3 to Level 5 
= Maintain Level 5 


= Solution: The grouping of practice areas and the relations among 
them that together address the problem for that context 


NORTHROP GRUMMAN 
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Software Product Lines 


Software Product Lines [IIIs iit 
Software Engineering Tectia Management Orginal Management 


What Building Blocks Do We Have? 


Architecture Definition Configuration Management Building a Business Case 
Architecture Evaluation Data pera ome nae and Customer Interface Management 
Component Development mig 1 alice Developing an Acquisition Strategy 
COTS Uthzaton Process. Definition Funding 
Mining Existing Assets. Scoping Launching and Instiutionalizing 
Requirements Engmeering Technical Planning Market Analysis. 
Software System Integration Technical Risk Management Operations 
Testing Tool Support Organizational Planning 
Understanding Relevant Domains Organizational Riek Management 
Structuring the Organization 
Technology Forecasting 
Training 
IDEAL Model 
= Stimulus for Change = Develop Recommendations — = Create Solution 
= Establish Context = Set Priorities = Test/Pilot Solution 
= Build Sponsorship = Develop Approach = Refine Solution 
= Charter Infrastructure = Create Solution = Install Solution 
= Characterize Current and "= Test/Pilot Solution = Analyze and Validate 
Desired State = Plan Actions = Propose Future Actions 


NORTHROP GRUMMAN 
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A Preliminary Set of CMMI Adoption Practice 
Areas 


|[Ciniatzing | Diagnosing | Establishing | Acting | Leaming 


Identify Need for deanaiviodel Identify Potential Develop Renee 
Change Improvements Infrastructure Pesdince: 
Scope : 
Identify Establish Pilot 

Bie emer Appraisal Goals Priorities Implementations Plan SCAMPI A 
Context 
Establish Select pone gil Conduct 

SPONsorsilp AU UIEL Se Infrastructure Improvements SOWIE 


Establish PeniGa Strategize Implement Conduct Causal 
Diagnosing A aa Project Organizational Analysis on 
Responsibilities ep Improvements Improvements Findings 


Identify Strategize Implement 
Complete PIIDs | Organizational Corrective 
BINS Cet Improvements Actions 
c Establish Implement 
Validate PIIDs Improvement Preventative 
Responsibilities — 
Conduct Gap __| Plan and Budget 
Appraisal Improvements 
Analyze 
Process | gpacurch 
Performance B Pp 
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Example - Transitioning from CMMI v1.1 to 
V1.2 


= Context: Organization that has achieved their CMMI v1.1 goals 


= Problem: Achieve similar goals against v1.2 quickly and cheaply 


pt 


Strategize 
Establish Project 
Improvement Improvements 
Responsibilities 


Implement 
Project 
Improvements Implement 


Corrective 
Plan and Budget Actions 


Improvements 


Establish 
Diagnosing 
Responsibilities 


Learn Model 
Strategize 
Organizational 


Improvements Implement 


Preventative 


Actions 
Implement 


Organizational 


Secure 
Improvements 


Sponsorship 


Reference: “A Practical Roadmap for Transitioning to CMMI v1.2”, NORTHROP GRUMMAN 


10 R. Hefner, 2006 CMMI Technology Conference and User Group 
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Example - Transitioning from Level 3 to Level 5 


= Context: Organization that has achieved CMMI Level 3 
Determine 


= Problem: Achieve Level 5, building on the Level 3 practices Lasts 
eadiness 


Strategize 
Project Plan SCAMPI A 
Improvements 


Establish 
Diagnosing 
Responsibilities Establish 

Improvement 

Responsibilities Conduct 


Implement SCAMPI A 
Project 


Identify Need for Improvements 
Change Conduct Causal 
Learn Model cenit |! Analysis on 
Strategize Findings 
Organizational 
scope Improvements Implement 
. cue Secure | Actions 
3 Sponsorship Implement 
Organizational Implement 
Identify Improvements oe ene 
Business Goals cues 
Reference: “A Practical Guide to Implementing Levels 4 and 5” NORTHROP GRUMMAN 


at (tutorial), R. Hefner, 2006 CMMI Technology Conference and User Group 
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Example - Maintaining CMMI Level 5 


=" Context: Organization that has achieved CMMI Level 5 


= Problem: Maintain Level 5 process performance in light of changing 
business goals 


- Pilot 
: Improvements 


Determine 
Appraisal 


Readiness 


Strategize 
Project 
Improvements 


Plan SCAMPI A 
Implement 


Project 
Improvements 


Analyze 
Process 
Performance 


Identify 
Business Goals 


Plan and Budget 
Improvements 


Conduct 


Strategize SCAMPI A 
Organizational 


Improvements 


Implement 
Organizational 


Reference: “Sustaining CMMI Compliance”, R. Hefner, Improvements PRTHROP GRUMMAN 


12 2006 CMMI Technology Conference and User Group 
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Conclusions 


= Patterns can be used to show how sets of improvement practices 
can be combined to achieve successful CMMI adoption 


; jt = Further work must be done to fully define the set of practice areas 
- | and the most common context-problem-solution sets 


NORTHROP GRUMMAN 
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Agenda 


» SHTECHNOLOGY = 
> Implementing processes for CMMI® Levels 2 and 3 
— Approach 
— Challenges 
> Measurement 
— Inthe Measurement and Analysis Process Area 
— Estimates and Actuals for projects 


> Benefits and Similar Observations 


4 


2 OFITECHNOLOGY = 


Implementing Processes 
for CMMI® Levels 2 and 3 


Approach 


» SHTECHNOLOGY = 


> Set up the process improvement activity as a project 
> Established the infrastructure first 

> Selected documentation format 

> Leveraged a strong set of existing practices 


> Rolled out defined processes to pilot projects with 
training for everyone 


A Project Approach 
» SHTECHNOLOGY = 
eObjectives 
eMonitoring and control strategy 
eRisks 
eResource estimates 
eMilestones 


eQuality assurance and 
configuration management 


«Schedule 


Regularly scheduled meetings 
*Delegate activities 
«Assignment of Action Items 
———_ *Follow-up 


eEstablish reasonable performance 
thresholds 

*Collect progress data at regular 
intervals 


wn les | Analyze results 


eTake corrective action 


The Process Infrastructure 


>» SHTECHNOLOGY = 


ee eT 
| 


eS 
soo ooo 0on 


Lal 


Documentation Format 


>» SHTECHNOLOGY = 


| IZIBOR Worktlog) Task(s) to be Performed Exit Criteria 
Entry 
Criteria 


Ee | : EPG CM determines need | 1. Process Change 


e for a pilot or training. Request Approved/Declined/Postpo 
— Approved : Review Process Change Form and ned PCR. 
Management Process Process Request and document work 

Change EPG Implementation instructions. 
& {age <=> Requests Decision. 

4. Process Hierarchy 
ea yes Architecture 
: Implement change 4. ETVX Diagram. 
tye - according to the 
Activity timeframe documented in 
HE no LJ task the PCR or by the EPG. 
(a=) Document Once the change is 
I succes Procedure complete notify PCR 


owner and the EPG. 


Process 


1,3 Publish in 


> Decision point 


Procedure 


©8H Technology LLC Proprietary and Confidential 
PRM-PD-02-1004 


Verification Steps 


e EPG configuration audits 
e EPG configuration status reports 


Existing Practices 


» SHTECHNOLOGY = 


> Estimating levels of effort 

> Project schedules 

> Vision documents 

> Defect tracking in an automated tool 

> Test scripts 

> Independent quality assurance organization 
> Version management for all documents 


Roll out of Defined Processes 
»* SHTECHNOLOGY = 
> Train pilot projects 
> Invite everyone 
> Rework as required 


. 


ra 
Sa 
ey 
oro 
| aN) 


Challenges 


» SHTECHNOLOGY = 


> Everyone wanted to be on all the process action 
teams 


> No one wanted to be a process area owner 


> Constraints on membership in the Management 
Steering Committee 


> Measures 

> Tailoring 

> Common practices 

> Lack of well defined chain of command for reporting 


New Words and Different Meanings 


SS) 3HTECHNOLOGY = 


> Project Plan is not equal to a project schedule 
> Validation can have many names 


> Configuration audits are not the same as process 
audits 


> Quality assurance is not testing 


> Product integration occurs, even if it is not a distinct 
procedure 


> Tailoring means flexibility, not a waiver 


Tailoring 


» SHTECHNOLOGY = 


> Define project size thresholds (value, FTE, or 
duration?) 


> How to record project tailoring selections 

> Document the guidelines to include roles and 
responsibilities 

> Notations on process or work product, optional, 
alternative practice or must use as-is 


Identifying Existing Practices 


» SHTECHNOLOGY = 


> Confusion: same words, different meanings 
> Confusion: different words, same meanings 
> Confusion: new words 

> Existing practices not fully documented 

> Existing practices not consistently practiced 


> Not all existing practices known across the 
organization 


Measures 


» SHTECHNOLOGY = 


> Benefit hard to see 
> We do enough of this already 
> No time; the contract doesn't pay for it 


> How can you estimate these things? (Quality cannot 
be predicted (estimated)) 


> Size does not mean estimated resource or level of 
effort 


> Staff level of effort is not the entire level of effort 


Measurement 


*& 


SHTECHNOLOGY = 


Measurement and Analysis Process Area 
»* SHTECHNOLOGY = 

> Corporate level Measures Guide 

> Four measurement categories 

> Measures worksheet 

> Industry best practices for thresholds 


> Repository on Sharepoint 


> Earned Value 10 84.36 
eee 


Measures Categories 


» SHTECHNOLOGY = 
> For the Measurement and Analysis Process Area 
— Level of effort 
— Size 
— Quality 
— Schedule 


ily 


Industry Best Practices 


»« SHTECHNOLOGY = 


Threshold for Analysis Corrective Action 


Effort 10% (Example: Cumulative labor time 1.Review original estimating basis and 
differs from the original estimate by determine impact on project schedule 
10% at any point in the project lifecycle) | and cost 
2.Assess impact on project risk 
3.Report to relevant stakeholders 


Quality 15% (Example: Number of cumulative | 1.Review original estimating basis and 
findings differs from the original determine impact on project schedule 
estimate by 15% at any point in the and cost 
project lifecycle.) 2.ASsess impact on project risk 

3.Assess impact on deliverables 
4.Report to relevant stakeholders 


Measures Worksheet 


>» SHTECHNOLOGY = 


Effort Estimates in Effort Project End Estimate 
stat Our Project Management 
Quality Assurance 


Product or Service 
Development 


Total Estimated 
Effort Sum of Estimated Effort 
Effort Actuals in Project Management 
Staff Hours : 

Quality Assurance 


Product or Service 
Development 


Total Actual Effort Sum of Actual Effort 
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Measure: Estimated Effort - Project A 
S, 
D 


®* 3HTECHNOLOGY = 


Jan Feb Mar Apr May Jun Jul 


20 


Measure: Actual Level of Effort - Project A 
S, 
™ S3HTECHNOLOGY = 


Jan Feb Mar Apr May Jun Jul 


—e— PM. QA Dev 
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Measure: Estimated Effort - Project B 


v 


&®* 3HTECHNOLOGY = 


FEB MAR APR - MAY — JUN JUL 


—e PM -#—QA -»- Dev 
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Measure: Actual Level of Effort - Project B 


4 


y a ~ SHTECHNOLOGY = 


FEB MAR APR MAY JUN JUL 


—e— PM —s—QA ~~ Dev 
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Measure: Estimated vs Actual Effort - ProjectA 


SS 3HTECHNOLOGY = 


8000 
7000 


6000 
9000 


4000 


3000 


2000 


1000 aa 


JAN FEB 


4, 
0. (2 ' 


MAR 


APR MAY JUN JUL 


—— PM-E—#—PM-A 


QA-E 


QA - A —*— Dev -E —@—Dev-A 
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Measure: Estimated vs Actual Effort Project B 


ve 


0 - SHTECHNOLOGY = 


—— PM-E -*-PM-A -+-QA-E QA -A -* Dev-E -*+Dev-A 
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Measure: 


Effort Final Results - Project A 


S\’ 6 
wae 
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13,646 


Feb 


Mar Apr May 


—¢e— Total Est —m— Total Actual 


Jun 


Jul 
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Measure: Effort Final Results - Project B 


& 


SHTECHNOLOGY = 


JAN FEB MAR APR MAY JUN 


—e— Cumulative Estimated Effort —t#— Cumulative Actual Effort 


JUL 
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Measure: Size 


» SHTECHNOLOGY = 


Estimated vs Actual Number of 
Requirements - Project A 


Jan. Feb. Mar. Apr. May Jun. Jul 


—e— Estimated Requirements —s— Actual Requirements 
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Measure: Quality 


» SHTECHNOLOGY = 


Estimated vs Actual Defects (Non-Cumulative) Project 
A 


100 


Jan. Feb. Mar. Apr. May Jun. 


—e— Estimated —s~— Actual 
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Measure: 


Quality 


» SHTECHNOLOGY = 


JAN 


Estimated vs Actual Defects - Project B 


FEB MA APR MAY JUN JUL AUG SEP 


—e— Estimated Defects —s#— Actual Defects 
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Measure: Schedule 


» SHTECHNOLOGY = 


Schedule Measures (Deliverables per month) 


Project A 
5 
4 
3 
2 
1 
O 2 5 
Jan. Feb. Mar. Apr. May Jun. Jul 
—¢e— Est Completions —s— On-Time Completions 


—. Late Completions —»<— Additional Unplanned Completions 


Pl Measures - Quality 


>» SHTECHNOLOGY = 


Actual Process Nonconformances (Non-Cumulative) from Audits 


vw 20 g 
oO 
c 18 | 17 
2 14 13 13 
5 12 | 
< 10 9 9 
Z 8 6 
5 6] 5 
a 4 
ae 
5 2} 000 000 000 : i a 000 joo 0 
Zz 0 | 
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 
B Actual ISO Major Nonconformances @ Actual ISO Minor Nonconformances 


O Actual CMMI Process Major Nonconformances Actual CMMI Process Minor Nonconformances 
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P| Measures - Effort 


>» SHTECHNOLOGY = 


Effort (Hours - Non-Cumulative) by PI Task 


Project PI Meeting PI Training PI Work PI Appraisal/Audit PI Consulting OT Management 
Management 


PI Tasks 


BG Jan @ Feb O Mar O Apr @ May G Jun B Jul O Aug @ Sep G Oct O Nov G Dec 
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P| Measures - Schedule 


>» SHTECHNOLOGY = 


Deliverable Status (Non-Cumulative) 


Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 


@ Estimated Completions & On-Time Completions Late Completions 


P| Measures - Size 


™ 
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Number of Items 


300 


250 


200 


150 


100 


50 


Number of Items in the PAL (Cumulative) 


Mar Apr May Jun Jul Aug Sep Oct Nov 


@ Cumulative Estimated Size @ Cumulative Actual Size 


Dec 


30 
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Benefits and Similar Observations 


Benefits and Similar Observations 


SS 3HTECHNOLOGY = 


> Implementation perspectives 
> Alternate Practices and Waived Process Areas 
> Things We Got Right 
> Lessons Learned 

> Benefits 


Implementation Perspectives 
ig SHTECHNOLOGY = 


Implement a Task 


More General 


-- Process Elements 


A1 Plan 
A2 Do the Work 
A3 Deliver 


A4 Close out the Task Organization 


Project 


Ie ak 
ee) ee 


Procedures 


Practices 


Sa 


More Detailed 


Process Decomposition 


Into Elements Modified from a briefing developed by the Software 
Productivity Consortium NFP, Inc. 
38 


Alternate Practices and Waived PAs 


SS 3HTECHNOLOGY = 


> SP 3.2 Perform Configuration Audits (CM) 

> Some Verification and most Validation performed by 
external group 

> Supplier Agreement Management 


> SP 1.3-3 Establish Product Integration Procedures 
and Criteria (Pl) 


Things We Got Right 


ig SHTECHNOLOGY = 
> Practiced what we preached 
> Set up the infrastructure before working the details 
> Leveraged existing processes 
> Engaged a consultant as objective third party 
> Used measures at all stages to determine progress 
and justify all requests for resources and support 
> Conducted a Class C, then a Class B before the 
Class A (with variation) 


Lessons Learned 


» SHTECHNOLOGY = 


> Focusing on measures is worth the effort 
> Including infrastructure early pays off 


> Manual works, but automated is much better (action 
items, change management, version control) 

> Consistent documentation standards make a 
difference in creating, documenting, tailoring, 
referencing and using processes 


Benefits 


» SHTECHNOLOGY = 


> Quality control improvements 


> Information sharing and communication among team 
members 


> Clarification of roles and responsibilities 
> Improved use of measures to make decisions 
> Significant additional business opportunities 


Quality Control Improvements 


» SHTECHNOLOGY = 


Average Number of Defects found in IV&V Before (Pre 
QC) and After (Post QC) Implementation of Quality 
Control Procedures 


40 
30 


Pre QC Post QC 
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Information Sharing 


» SHTECHNOLOGY = 


> Diversity on the process action teams 

> Cross-organization membership on the Engineering 
Process Group including ISO organizations 

> Cross-organization membership on the internal 
process audit teams 


Clarification of Roles and Responsibilities 


SS» 3HTECHNOLOGY = 


> Roles and Responsibilities defined in a matrix 


> Combined separate tasks by process/lifecycle phase, 
roles and process tailoring by size 


> Created summary cards for certain roles 


Roles and Responsibilities Matrix 


» SHTECHNOLOGY = 
Project Duration, Size andfor £23 months or $150,000 or ¢ 5 FTE staff resources and Risk and Criticality to Client Operations iz 
Value Threshold: Low or Medium 
Governing Statement: Those rezpongible for creating work products must encure that appropriate staff provide input, review and approve content Role and Responsibility 
Process[P] Clue Date (if applicable, 
or Process unless overridden by Q@ C Senior 
Process or Process Artifact | Artifact{A) Comment the contract) Relited Resources PH oTH WOM Mgnt 
Pre-d ard 
We Taioring or dferaste 
Practice Cousin 
Optional 
At Risk Form 4 Used when funds are not available but work is required, c Bia 
Shutement of Work or AFF 4 Customertclient generates this document NA Fi PB oR RE BE 
SOWIRFP Proposal (including cost 
proposal] 4A By due date c | I] | A 
Vigion Dacument A C RF RAR 4 
Prodect 
Propect Management Process 
Wa Fatoring or aitereste foment 


This iz accomplished by the Project Setup Form. This can 
be delegated to PM: for pre-award work under $200k; iz 
optional for work defined under existing TEM, FFP 

contracts and maintenance. Form goes to Finance. The At 
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Improved Use of Measures to Make Decisions 


» SHTECHNOLOGY = 


> Decision Analysis for selecting technical 
implementation options, pilots and training for 
process change requests 


> Setting measurement goals to guide analysis efforts 
> Risk management 
> Escalation of process audit results 


Questions 


>» SHTECHNOLOGY = 


rep fe 


It's QUESTION TIME?! 


2006 State of Software 
Measurement Practice Survey 


NB) FAW@7\V/| V/s =Yo1 a] ale) (ele hVarclare Mm Olsy>1 6s 
Conference 


WHF a,@l aecksieral(e 
INYo)¥{=100] 01>) am ko Par-4010 16) 


Software Engineering Institute | Carnegie Mellon © 2006 Carnegie Mellon University 


Agenda 


Introduction 
¢ Survey objectives & approach 
¢ The population being studied 
¢ Sampling plan 


Results 
- Response rates and outcome 
¢ Population demographics 
- Attitudes and beliefs about measurement use 
- Measurement guidance that is used 
¢- Measures that are reported 


Summary Observations 


2006 State of S/W Measurement Survey 


—— Software Engineering Institute | CarnegieMellon Mark kasunic, 11-15-2006 


© 2006 Carnegie Mellon University 


Survey Objectives 


The objectives of this survey are to characterize 


¢ the degree to which software practitioners use 
measurement when conducting their work 


¢ the perceived value of measurement 


¢ approaches that are used to guide how measures are 
defined and used 


¢ the most common types of measures used 
by software practitioners 


2006 State of S/W Measurement Survey 


—— Software Engineering Institute | CarnegieMellon Mark kasunic, 11-15-2006 
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Characteristics of the Survey 


We used a structured, self-administered questionnaire that was available both via the 
World Wide Web and in paper form. 


The questionnaire was designed to be short (17 questions) and easy-to-complete with 
questions phrased in close-ended format. Several questions allowed for short open- 
ended responses. 


Stratified random sampling was used to select candidate respondents from a population 
comprised of members from three different subpopulations. 


Candidate respondents were offered incentives to participate including 
¢ platinum membership to the Software Engineering Information Repository 
(SEIR) that provides access to documents otherwise unavailable through regular 


membership 


¢ early access to the survey results 


2006 State of S/W Measurement Survey 


== Software Engineering Institute | CarnegieMellon Mark kasunic, 1-15-2006 
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The Population Being Studied 


The population that we would have /iked to have studied is the entire 
existing body of software practitioners in the world. However, such a 
representative database was unavailable to us. 


The population that we did use for this study included individuals who: 


4) were entered into the SEI customer relations database during 
2004-2005 


registered to gain access to the SEl’s Software Engineering 
Information Repository (SEIR) during 2004-2005 


2006 State of S/W Measurement Survey 


—— Software Engineering Institute | CarnegieMellon Mark kasunic, 11-15-2006 
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Important to Remember When Interpreting Survey 
Results 


SEl 
SEIR Customer 
Registrants Relations DB 


49.7% 


These are 
not the same 


Population of all * SEI 
software practitioners Survey results can not be generalized Memb 
beyond the population used in this study. eel 


—— Software Engineering Institute 


. 2006 State of S/W Measurement Survey 
Carnegie Mellon Mark Kasunic, 11-15-2006 
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Sampling Plan 


Population 
Subpopulation Size 
Customer Relations 6,398 
SEI Members 1,242 
SEIR registrants 7,040 

Total 15,180 
Calculated for: 
precisionof £25% 
confidence of 95% 


== Software Engineering Institute 


Adjusted 
Sample Size 


Sample 
Size 


2010 


1,242 


2040 


9,292 


Adjusted based on estimated 
30% response outcome. 


age responses 
Ineligible respondents 


2006 State of S/W Measurement Survey 
Mark Kasunic, 11-15-2006 


© 2006 Carnegie Mellon University 
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Response Outcome Rates 


Standard Definitions 


Final Dispositions of Case Codes 


and Outcome Rates for Surveys RRI1 — 4) A % 


RDD Telephone Surveys 
In-Person Household Surveys 
Mail Surveys of Specifically Named Persons 
Internet Surveys of Specifically Named Persons 


RR2 = 50.7% 


2006 


THE AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH 


http://www.aapor.org/pdfs/standarddefs_4.pdf 
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84 Countries Represented 


Respondents by Continent 


319 
225 


35 


33 
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Survey Respondents 


Other 634 
Project manager 340 
Engineer 250 
Executive manager 227 
Program manager 225 
Analyst 152 


Programmer 67 


0 100 200 300 400 500 600 700 
Frequency 
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Approximate Population Proportions 


DoD & Government 


Commercial 
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Agenda 


Results 


Response rates and outcome 
Were subpopulations different? 
Population demographics 
=> Aititudes and beliefs about measurement use 


How are you involved with measurement? 

Are purposes for measurement understood? 

Does measurement help? 

ls measurement used to understand product/service quality? 
Documented measurement processes ? 

Measurement definitions understood and consistent? 

Do measurable criteria exist for products and services? 

Is corrective action taken when thresholds are exceeded? 


Measures that are reported 
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Involvement With Measurement 


Provider 290 
7 32.4% 
User 324 
Both a provider and a user 1142 


¢ Don’t do measurement. 


Other 136 
¢ /set up measurement programs. 


0 200 400 600 800 1000 1200 #1400 
Frequency 
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Provides (only) or Uses (only) 


40.0% 


Uses measurement data but does not provide it to someone else. 
Provides measurement data but does not use it. 


30.0% 


20.0% 


Percent 


10.0% 


0.0% 


Exec Prog. Project Engineer Analyst Programmer Other 
Manager manager 
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Purpose for Measuring Is Understood 


Frequently 
Occasionally 
Rarely 
Never 


| don’t know 


N/A 


69 


390 


200 


400 


600 800 1000 1200 1400 
Frequency 
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Believe That Measurement Helps 
(To Some Degree) 


Agree: 


92% 
Disagree: 2% 


~~~ Not sure: 4% 


2006 State of S‘W Measurement Survey 


—— Software Engineering Institute | CarnegieMellon —_ marc kasunic, 11-15-2006 


© 2006 Carnegie Mellon University 


Measurement Used to Understand Quality of 
Products & Services 


Frequently 


Occasionally 


Rarely 210 1 


Never 39 


| don’t know | 17 


N/A 


0 200 400 600 800 1000 
Frequency 
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Documented Process for Collecting 
Measurement Data 


Frequently 
Occasionally 559 
Rarely 
Never 87 
| don't know 18 
0 200 400 600 800 1000 
Frequency 
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Measurement Definitions Are Understood & 
Consistent 


fea Aether aaa nasehasede Deceetas N/A: OX, 
10% 


24% 


1868 Responses 


Not sure: 4% 
—— Software Engineering Institute 


Carnegie Mellon 
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sity 


Measurable Criteria Exist for Products & 
Services 


Frequently 


Occasionally 711 


Rarely 
Never 


| don’t know 


N/A 


0 200 400 600 800 1000 
Frequency 
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Corrective Action Taken When Measurement 


Threshold Exceeded 


Frequently 
Occasionally 


Rarely 


Never 


| don’t know 


N/A 


Frequency 
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Action-Oriented Response to Measurement 
Information 


60 


1847 Responses 


50 - 


40 


Percent 
30 


20 —— —| — — —— ——| — 


0 I | 
Program Other Executive Project Engineer Analyst Programmer 
Manager Manager 


Measurable criteria established (frequently) 
Corrective action taken when threshold met (frequently) 
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Agenda 


Introduction 
¢ Survey objectives & approach 
¢ The population being studied 
¢ Sampling plan 


Results 
- Response rates and outcome 
¢ Population demographics 
- Attitudes and beliefs about measurement use 
=~ Measurement guidance that is used 
- Measures that are reported 


Summary Observations 
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yz.) 


Measurement Methods Used 


CMMI M&A Process Area 1045 55.9% 


Goal-Driven Software Measurement 383 20.5% 


GQM 354 

Other 219 Gi7%) 
| The percentages do not add to 

PSP/TSP 210 > 100% because some individuals 
| use more than one method. 
PSM 171 9.2% 
ISO 15939 139 <> 

0 200 400 600 800 1000 1200 


1868 Responses Frequency 
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“Other” Methods Used 


Home-Grown 


No Metrics (comment) 


Proprietary 
Other 


Customized from Other 


CMM 


Standards (IEEE, ISO, etc.) 
Six-Sigma 
ITIL 


Balanced Scorecard 


By Mandate 


SPC 


Function Points 


39 
32 
25 
25 
20 
19 


15 


219 Responses 
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Those Using “CMMI M&A Process Area” 


600 
# Total using other method 
511 # Using CMMI M&A and other method 
500 - 
400 - ' 


Frequency 
aw 
i) 
je) 


Boo 238 
200 - . 
127 115 
100 - os 89 
0 ! ! ! 
CMMI GQM Goal- PSM PSP/TSP ISO 15939 Other 
(only) Driven 
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¢ Survey objectives & approach 
¢ The population being studied 
¢ Sampling plan 
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=~ Measures that are reported 


Summary Observations 
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Measurements that Are Reported 


Risks identified 


Schedule progress 90 


Code growth 

Defects removed 
Defects identified 
Effort applied to tasks 


Capability/requirements stability 


Do not report 
Do report 


== Software Engineering Institute 


156 1461 
1639 
490 822 
27 1329 
158 1435 
inh) 1535 
401 1117 
0 200 400 600 800 1000 1200 1400 1600 1800 


Carnegie Mellon 
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Introduction 
¢ Survey objectives & approach 
¢ The population being studied 
¢ Sampling plan 
Results 
- Response rates and outcome 
¢ Population demographics 
- Attitudes and beliefs about measurement use 


- Measures that are reported 


=> Summary Observations 
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Influence of Role on Response - 1 


In general, there were significant differences in response patterns when 
comparing management versus staff. 


Management Staff 
Executive Engineer 
Program Manager Analyst 
Project Manager Programmer 


Statistical tests of significance demonstrated that the differences were 
significant with confidence of at least 99% in all cases (and 99.9% in 
some cases. 


* Hypothesis test for equality of proportions 
¢ Chi-Square test for significance 
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Influence of Role on Response - 2 


When compared to staff, management responded more strongly that 
¢ they understand the purposes for measurement 
* measurement helps their team perform better than without it 


- they use measurement more often to understand the quality of their 
products and services 


¢ they follow a documented process more often for collecting and 
reporting measurement data 


* measurement definitions are commonly understood and consistent 
in their organization 


¢ measurable criteria exist for their products and services 


¢ corrective action is taken when a measurement-based threshold has 
been exceed 


In general, the differences are statistically significant. 


2006 State of S‘W Measurement Survey 


—— Software Engineering Institute | CarnegieMellon mark kasunic, 11-15-2006 33 


© 2006 Carnegie Mellon University 


Influence of Organizational Size - 1 


Number in Organization 
<100 101-499 2500 


Using measurement-based data helps my team to 
perform better than without using it. «18.4% 81.5% 86.8% 


* Percent that Agree or Strongly Agree. 


There exist measurable criteria for the products 37.0 46.4 54.7% 
and services to which | contribute. 


_ | use measurement to understand the quality of 


_ the products/services that | work on. 35.1% 41.1% 46.2% 


>> » 


: My team follows a documented process for 
collecting measurement data. 65.7% 71.6% 72.1% 


* Percent that responded, “Frequently” to the listed questionnaire item. 
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Comparing Industry to Government 


Industry Government 


Using measurement-based data helps my team to 


perform better than without using it. 84.5% 80.0% 


Definitions of measures used in my organization are 
commonly understood & consistent. 37.1% 31.9% 


* Percent that Agree or Strongly Agree. 


Differences statistically significant with confidence 95%. 
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Comparing USA to Other Countries 


USA Other 
Using measurement-based data helps my team to 
perform better than without using it. 80.1% 85.9% 
Definitions of measures used in my organization are 
commonly understood & consistent. 31.3% 42.4% 


* Percent that Agree or Strongly Agree. 


Differences statistically significant with confidence 99%. 
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Using Measurement to Improve 


It is notable and a bit alarming that only 40.3% of all respondents reported 
that corrective action is taken when a measurement threshold has been 
exceeded. 


Close to 20% of respondents reported that corrective action is rarely or 
never taken when a measurement threshold is exceeded. 


Measurement doesn’t help much Plan 


unless the information is acted upon. 
i * 
&. 
Measure 
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To be published this year: 


BMalsmeliclismelmeyeyiiic-lasmiststielaslaatsvall 
Practice: Results of 2006 Survey 


TECHNICAL REPORT 


CMU/SEI-2006-TR-009 
ESC-TR-2006-009 
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Agenda —— 


e Main Message: 


e CMMI-SVC is a minimal and Jogical extension to CMMI 
v1.2 content, allowing current CMMI users to reuse 
CMMI investments to improve service performance 


e Agenda: 
e CMMI-SVC’s minimal model footprint 
e illustrating the substantial reuse of CMMI-DEV material 


e CMMI-SVC’s /ogical construction 
e an introduction to CMMI-SVC model content 
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Why Services? 


Nov 15, 2006 


e Service society 
e Customer discontentment 
e Legislation 


e Government and industry 
trends 
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Complementary CMMI 
Constellations 


CMMI-DEV provides 


guidance for CMMI-SVC provides 
measuring, guidance for delivering 
monitoring and services within 
managing organizations and to 
development CMMI-SVC external customers 
processes 
CMMI-ACQ 
provides guidance 
for enabling 
CMMI-DEV CMMI-ACQ informed and 
decisive 
acquisition 
leadership 
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Reusable CMMI-SVC a 


| Service Modifications: 
' ¢ 21 amplification in 7 PAs 
' e 5 added references } 
1 modified PA(REQM) : 
¢ 1 specific goal } 
¢ 2 specific practices 


% of CMMI-DEV PAs are reused; 
% of Corporate Investments are potentially reusable! 


What Makes Service Seana? 
Different? —_— 


e Aservice is an intangible, non-storable product 


e Services can exhibit great variability regarding 
e Services requested (both number and type) 
e Incidents encountered 
e Resources needed (for a single request, over time) 


e Disruptions encountered (discontinuities including 
upgrades) 


e Quality of the services provided 


Nov 15, 2006 CMMI Technology Conference, Denver CO 5 


CMMI Levels Applied to 
Services 


Pa 
Cmmi 
gil 


e Mature service management prioritizes its 
improvements into 5 levels: 


i. 
2 
3. 


Current, as-is services 
Basic service management and support discipline 


Organizational consistency of level 2-3 practices, 
including service delivery, and advanced management 
practices 


Quantitative predictors of key service qualities 
Optimization of key service qualities 
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cmmi 
Level 2 CMMI-SVC Practices _~ 


e At level 2, the maturing services organization adds 
to the foundational process areas: 
e Service requirement practices 
e Service request and incident handling 
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Service Requirement 


SNe 
; CcCmmi 
Practices —_— 
e Service requirement agreements are an important 
building block for effective service management. 


e They specify terms and conditions for: 
e Provided services 
e Metrics 
e Service levels 
e Liabilities 
e Actions in specific circumstances 


From http://www.iec.org/online/tutorials/service_level/ 
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, Ae 
. CMMI 
Requirements Management -_~ 


e To require Service Requirements Agreements, the 
CMMI-SVC provides: 


e Requirements Management SVC (REQM) 

e Extends the foundational REQM to include the establishment and 
maintenance of written agreements between service providers 
and customers on service requirements and service levels. 

e The CMMI Services team chose to add a goal and two practices 
to REQM rather than a separate PA to reuse the requirements 
paradigm and to keep the model footprint smaller. 
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Handling — 


e Based upon the Service Requirement Agreement, the 
customer requests services and reports incidents. 


e Service request - request from a customer to deliver (part of) a 
previously agreed-upon service. 


e Incident - an interruption or potential interruption to the agreed level 
of service. 
e A mature organization manages requests and incidents in 
an orderly fashion. 


e CMMI-SVC provides: 


e Incident and Request Management PA - to ensure the timely 
resolution of requests for service and incidents that occur during 
service delivery 
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Level 3 CMMI-SVC Practices _~ 


e At level 3, the maturing service organization adds 
to the foundational process areas: 
e Advanced project management practices 
e Capacity and availability management 
e Service continuity management 
e Advanced support practices 
e Problem management 
e Service establishment and delivery practices 
e Predicated on the service system 
e Process management practices 
e Catalog of standard services and service levels 
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Advanced Project As 
Management Practices 4 


e CMMI-SVC provides: 


e Capacity and Availability Management (CAM) 


e Purpose: to plan and monitor the effective provision of resources to 
support service requirements 


e Addresses the resources necessary for Supporting service request and 
peak demand, as well as service project tasks. 


e Based on measurements collected during level 2 activities 
e Service Continuity Management (SCON)* 


e Purpose: to establish and maintain contingency plans for continuity of 
agreed services during and following any significant disruption of normal 
operations 


e Addresses the actions in specific circumstances as laid out in the service 
requirements agreement 


* optional process area (independent named service addition) 
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cmmi 
Advanced Support Practices —~ 


e CMMI-SVC provides: 


e Problem Management (PRM) 


e Purpose: to prevent incidents from recurring by identifying and 
addressing underlying causes of incidents 

e Dynamically and proactively handles known problems 

e Problem - a situation in the service system that is the underlying 
cause of incidents. All incidents have one or more underlying 
causes, regardless whether the service provider is aware of the 
Cause or not. 
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service System —_— 


e Service System is a necessary building block for 
understanding the effective delivery of services 


e Service System — An integrated and 
interdependent combination of components, 
consumables, and people that satisfies service 
requirements. 

e Aservice system encompasses everything required for 


service delivery, including work products, processes, 
infrastructure, consumables, and customer resources. 


e The quality of a service is often dependent upon the 
quality of the service system. 


e Aservice system should be developed in a mature way. 
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Service Establishment and cmmi 


Delivery Practices a 


e The CMMI-SVC provides for: 


e Service Transition (ST) 


e Purpose: to deploy new or significantly changed service systems 
while managing their effect on ongoing service delivery 


e Service Delivery (SD) 


e Purpose: to deliver services in accordance with service 
agreements 


e Service System Development (SSD)* 


e Purpose: to analyze, design, develop, integrate, and test service 
systems to satisfy existing or anticipated service agreements 


* optional process area (independent named service addition) 
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Process Management 


TONe 
cmmi 
Practices = 4 


e Mature service management uses: 
e Catalog of standard services and service levels 


e The CMMI-SVC provides for: 


e Organizational Service Management (OSM)* 


e Purpose: to establish and maintain standard services that ensure 
the satisfaction of the organization's customer base 


* optional process area (independent named service addition) 
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CMMI-SVC Process Areas 


Process Management 


e Organizational Innovation and 
Deployment (OID) 


e Organizational Process Definition (OPD) 

e Organizational Process Focus (OPF) 

e Organizational Process Performance 
(OPP) 


e Organizational Service Management 
OSM)* 


e Organizational Training (OT) 


Service Support 

Causal Analysis and Resolution (CAR) 
Configuration Management (CM) 
Decision Analysis and Resolution (DAR) 
Measurement and Analysis (MA) 
Problem Management (PRM) 


Process and Product Quality Assurance 
(PPQA) 


* Service Addition (optional) 
Nov 15, 2006 


Service Establishment and Delivery 


Incident and Request Management 
(IRM) 


Service Delivery (SD) 
Service System Development (SSD)* 
Service Transition (ST) 


Project Management 


Capacity and Availability Management 
(CAM) 


Integrated Project Management (IPM) 
Project Monitoring and Control (PMC) 
Project Planning (PP) 

Requirements Management (REQM) 
Risk Management (RSKM) 

Quantitative Project Management (QPM) 


Service Continuity Management 
(SCON)* 


Supplier Agreement Management (SAM) 
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Process Area Staging 


Process Area 


Capability and Availability Management (CAM) 
Incident and Request Management (IRM) 


Organizational Service Management (OSM)* 


3) 
2 
a 
Problem Management (PRM) P83 | 
Service Continuity Management (SCON)* p38 
Service Delivery (SD) a 
Service System Development (SSD) * LT. 

Ps 


(Used in place of Development Model for small organizations) 


Service Transition (ST) 


* Optional process areas (independent named additions) 
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Practices 
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18 


LONe 
cCmmi 
Conclusion gt 


e CMMI-SVC is a minimal and logical! extension to 
CMMI v1.2 content, allowing current CMMI users 
to reuse CMMI investments to improve service 
performance 
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: Cmmi 
Appendices or 
e Industry Participation 
e Schedule 
e References 


Authors & Contributors 
Referenced Service Models or Standards 
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Industry Participation 


e Visit the public Services CMMI web pages 
e CMMI web page - http://www.sel.cmu.edu/cmmi/ 
e CMMI for Services Public Workspace - 
http://oscw.sei.cmu.edu/pub/bscw.cqgi/0/424939 
e Participate in pilot and review period for the draft 
CMMI for Services (tentatively Nov 06 - Feb 07) 
e Submit change requests to draft release 


e Appraise your organization internally against the draft 
model and submit appraisal questionnaire 
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Tentative Schedule 

e Oct-Nov 2006 Steering Group review 

e Nov-Feb 2007 Piloting to refine practices 

e Mar-May 2007 Updates written and reviewed 

e Jun-Aug 2007 OA and final review 

e Sep 2007 Release of CMMI-SVC Product Suite 
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References 


e CMMI: The New Architecture - 


http://www.sel.cmu.edu/news-at- 
sei/columns/cmmi-in-focus/2006/05/cmmi-in-focus- 


2006-05.htm 


e CMMI for Services Architecture — to be released 
with draft CMMI for Services model 
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Referenced Service Models Seana? 
or Standards SY 


e IT service models and standards included in 

mapping and gap analysis: 

e Information Technology Infrastructure Library (ITIL) 

e British Standard 15000: IT Service Management (BS 
15000) 

e Control Objects for Information and related Technology 
(COBIT) 

e Information Technology Services CMM (ITSCMM) 
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CMMI Implementation in 
Manufacturing Processes 


SARIT ASSARAF 
ITZHAK LAVI 


ISRAEL AIRCRAFT INDUSTRIES LTD. 


CMMI Conference —Denver Nov. 2006 


CMMI Implementation In Manufacturing Processes dt I. LAVI S. ASSARAF — IAI Nov.2006 


israel Aircraft Industrie 


@ Largest industrial company in Israel 
@ 15000 employees, 2.3 BS annual sales 


@ Involved in Development, Production, Maintenance 
and Service of Aerospace Systems 


@ A large part of the annual sales are from Manufacturing, 
Maintenance and Service activities 


@ IAI divisions are ISO9000 and AS9100 certified 


@ Started SW-CMM based development process 
improvement efforts in 1992 


@ Started CMMI based efforts in 2002 
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Gi up 
STRIES LTD 


israel Aircraft Industrie 


i 
ELTA Space & Diilittarry 


Missiles; Avlinc:ranfits; 
She MALAT 


ai 


SIGINT, EW and 
Communication 
Qretta 
Technologies _ 


Significant manufacturing activity 
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S) LAHAV DIVISION 
Military Aircraft Group 
ISRAEL AIRCRAFT INDUSTRIES LTD 


Corporate Initiatives for Process Im 


orovement 


Soy ip Crees AV oO lanl leo enre ale (@AN)) 
> Production & Manufacturing Process improvement 
FS Nee irsition Press lilo oniaaie 


e Increase 
Quality 


system & Software Development Processes and } 
Mixigteyele)[eYe/ [5 _— * Increase 


Sales 


New Product Introduction (NPI) ¢ Increase 
IAI Uniform Project Process Framework Profit 


LAHAV DIVISION " 
© Military Aircraft Group PSSET fF 
ISRAEL AIRCRAFT INDUSTRIES LTO tae § oer 


Israel Aircraft Industries Process Improvement Path -1 


Starting activities for Starting CAI 
project management activities and NPI 
process improvement development 
1993 1999 


Starting SPIP - CMM Identifying System Starting ASSET; CMMI 
based Software Process Engineering process __ eekise| ys Opis iat 
Improvement Program needs and starting 5 iM OLOVS i 
1992 SYSPIP 1997 


AVI I 
Ann 
2000 
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@ Process improvement efforts are coordinated at the corporat 
level and sponsored by the company VP of Operations | 


@ Many infrastructure assets are developed at the corporate level 
and are adapted by the groups and divisions. Other assets are 
developed within the groups and divisions 


@ Process improvement implementation is conducted at the group 
and division level and sponsored by the group or division 
management 


@ Initially, CMMI based Process Improvement efforts were 
focused on Development Processes, while CAI 
(Competitiveness Ability Improvement) efforts were focused 
mainly on Manufacturing Processes 
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ELTA 


hillary, Conmmencial BEDEK 


Space & 


Missiles Aline ranfts Aiinc rays 

ELTA Business Engines 

IMINT and Radar el Aircrafts * 
MBT ~ Aerospace Components 

ELTA Technologies Assemblies * 
SIGINT, EW and 
Communication TAMAM © Production 

Technologies 
ELTA 
Technologies oe he RAMTA 


Hl CMMI Based Process Improvement Activities 
HE significant manufacturing activity 
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l Jsing CMMI for Manufacturing Process | 
8 CAI provides a collection of techniques for é attaining 
and Productivity while CMMI provides a frame ework fi 
integrated Process Improvement activities in an organization 


sr 


@ Most (if not all) CMMI Process Areas seem to fit Manufacturing as 
well as Development 


@ Project Management Process Areas 

@ Process Management Process Areas 

@ Support Process Areas 

@ Integrated Product and Process Development 
@ Engineering Process Areas (?) 


@ IAI decided to include manufacturing processes in CMMI 
implementation scope in one of its divisions - LAHAV 
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Applying CMMI in Manu 
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and Support Process Areas in Manufacturing Is 
obvious 


@ Question: 


@ How should Engineering Process Areas be interpreted and 
implemented in Manufacturing? 


@ Answer: 


Production Engineering, including production infrastructure 
development 


@ Addressing problems and managing changes in products 
and production line 
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@ Build To Spec (BTS) 


@ The product specification is provided by the customer and 
the project has to design the product, design and build the 
production line, and manufacture products according to 
customer order 


@ Build To Print (BTP) 


@ Complete product design is already available — either 
supplied by the customer or from previous BTS project, 
and the project (Sometimes) has to design, build or change 
the production line infrastructure, and manufacture units 
according to customer order 
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Build to Spec (BTS) Projects 


> We Customer Requirements are defined r main ly | 
Specification 


@ Based on Product Specification, the product is designed, 
integrated and tested. The production line requirements are 
developed, and the production infrastructure is designed, 
integrated and tested 


@ The “developed product” includes not only the product to be 
manufactured, but also the production line infrastructure 


@ CMMI implementation is focused on both product design and 
Production Engineering Processes 


@ The production line infrastructure may serve manufacturing of 
several products in several production projects 
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| Build to Spec 
@ RD SP1.1: Elicit Needs 


® Analysis of Product Specifications and Staxeholdai needs ar 
@ RD SG3: Analyze and Validate Requirements 

@ Simulation of Product Design and the Production line 
@ TS SP1.2 + RD SP3.1: Operational Concepts and Scenarios 

@ Product Operational Scenarios and Product Line Production Scenario 
@ TS SP3.2: Develop Product Support Documentation 

® Includes Production Line Support Documentation 
@ VAL: Validate the Product 

@ Validation of the product prototype & production line using predefined scenarios 
@ PP and PMC: Plan and Monitor the Project 

@ Product Engineering Management and Mass Production Management 
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ee dees Applying CMMI in Manufacturing 

Build to Print BTP Projects 

@ Customer Requirements are defined mainly i in terms of 
Product Technical Package 


% Drawings, Part Lists, etc. 


@ Based on Product Technical Package, the production line 
requirements are developed, and the production 
infrastructure is designed, integrated and tested 


@ The “developed product” is actually the production line 
infrastructure, and CMMI implementation is focused on 
Production Engineering Processes 


@ The production line infrastructure may serve manufacturing 
of several products in several production projects 
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: Build to Print (BTP) Proje: 
@ RD SP1.1: Elicit Needs | | 
@ Analysis of Product Design and Stakeholders needs and expectations 


@ RD SG3: Analyze and Validate Requirements 
@ Simulation of the Production line 
@ TS SP3.2: Develop Product Support Documentation 
@ Includes Production Line Support Documentation 
@ PI SP3.4: Package and Deliver 
@ Packaging and delivering the product 
@ VAL: Validate the Product 
@ Validation of the production line using pre-defined scenarios 
@ TS SP1.2 + RD SP3.1: Operational Concepts and Scenarios 
@ Product Line Production Scenarios 
@ PP and PMC: Plan and Monitor the Project 
@ Product Engineering Management and Mass Production Management 
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Lessons Learned 


@ CMMI can be applied to manufacturing projects 


@ Usually all Process Areas are applicable and can appropriately be 
interpreted for manufacturing processes 


@ When some Process Areas seem to be non applicable for 
manufacturing, the Continuous Representation can be used 


@ Although manufacturing processes are very different from 
development processes, they are still part of the division set of 
standard processes 


@ When manufacturing activities represent a significant portion of 
the activities in the division, they should be included in the 
CMMI implementation and appraisal scope, due to their 
significant contribution to the division’s business 
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natural to implement in manufacturing, such #s. 


Measurement and Analysis 
Causal Analysis and Resolution 
Quantitative Project Management 
Organization Process Performance 
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Recommendations 


@ When an organization starts to implement CMMI 
practices and plans CMMI process appraisals, it should 
not refrain from including “non developmental” 
processes, like manufacturing, in the scope, especially 
when these processes are responsible for a significant 
amount of the organization’s revenue 


@ In BTP projects the production line processes should be 
evaluated as well as the manufactured product 
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Purpose 


The purpose of this talk is to provide an overview of 
important appraisal concepts. 
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Topics 

Appraisal Requirements for CMMI 
SCAMPIS™ Appraisal Methods 
Fundamental Concepts 


Becoming a Lead Appraiser or Team Leader 
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Appraisal Requirements for CMMI 


The Appraisal Requirements for CMMI (ARC) defines the 
requirements considered essential to appraisal methods 
intended for use with CMMI models: 
¢ based on appraisal principles common to source 
methods 
¢ defines three classes of appraisal methods that reflect 
common usage modes of appraisal methods 


ARC requirements are allocated to each method class to 
align with usage mode characteristics. 
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ARC Appraisal Principles 

Start with an appraisal reference model (e.g., CMMI for 
Development). 

Use a formalized appraisal process (e.g., SCAMPI A). 
Involve senior management as the appraisal sponsor. 
Focus the appraisal on the sponsor’s business objectives. 
Observe strict confidentiality and non-attribution of data. 
Approach the appraisal collaboratively. 


Focus on follow-on and decision-making activities by 
producing actionable appraisal results. 
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SCAMPI Appraisal Methods 
Fundamental Concepts 


Becoming a Lead Appraiser or Team Leader 
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SCAMPI Family: 3 Classes of 
Appraisal Methods 


breadth of tailoring 


SCAMPI C provides a wide range 


of options, including characterization 

of planned approaches to process 

implementation according to a scale 

defined by the user. depth of 
investigation 


SCAMPIB provides options in model 
scope and organizational scope, but 
characterization of practices is fixed 
to one scale and is performed on 
implemented practices. 


SCAMPI A Is the most rigorous 
method, and is the only method that 
can result in ratings. 
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Differentiating Attributes For 
Appraisals 


The three key differentiating attributes for appraisal classes 
are: 

¢ degree of confidence in the appraisal outcomes 

e generation of ratings 

° appraisal cost and duration 
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SCAMPI Requirements 


Types of Objective Documents Documents | Documents 
Evidence Gathered and interviews | and or interviews 
interviews 


Ratings Generated Goal ratings No ratings No ratings 
required allowed allowed 


Organizational Unit Required Not required | Not required 
Coverage 
Appraisal Team Leader | SCAMPI A SCAMPI B SCAMPI B 


Requirements lead appraiser |andC team | andC team 
leader leader 
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Selecting Appraisal Methods 


Consider a family of appraisal methods in determining 
overall appraisal needs: 
e Fit the appraisal method to the actual need. 
e Many appraisal needs can be met without maturity level 
or capability level ratings. 


SCAMPI A may not be the most appropriate choice for 
organizations early in their process improvement cycle. 
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Fundamental Concepts 


Appraisal 

Appraisal reference model 
Organizational unit 
Organizational scope 
Instantiation 

Focused investigation 


Objective evidence 
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Appraisal 


An examination of one or more processes by a trained 
team of professionals using an appraisal reference model 
as the basis for determining strengths and weaknesses. 
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Appraisal Reference Model 


The CMMI model to which an appraisal team correlates 
implemented process activities. 
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Organizational Unit 


The part of an organization that is the subject of an 
appraisal. An organizational unit deploys one or more 
processes that have a coherent process context and 
operates within a coherent set of business objectives. An 
organizational unit is typically part of a larger organization, 
although in a small organization, the organizational unit 
may be the whole organization. 
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Organizational Scope 


The collection of projects and support functions that 
provides instantiations of practices used within, and 
representative of, an organizational unit. 
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Organizational Scope Determination ~ 


Samples in the organizational scope must be one of the three types 

below: 

Focus projects (FP): must provide objective evidence for every PA 

Non-focus projects (NFP): must provide objective evidence for one 

or more PAS 

e Support Groups: must provide objective evidence for practices 
which address organizational infrastructure or functions 


lf the model scope includes project related PAs, the Organizational 
Scope must contain at least one FP 


If the OU includes more than 3 projects, the Organizational Scope 
must include sufficient FP & NFP to generate at least 3 instances of 
each practice in each project-related PA in the model scope of the 
appraisal 


Note: The MDD is silent on OUs with 1-3 projects. We recommend the 
Organizational Scope include all projects as focus projects. 
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Organizational Scope Description 


Critical factors that influence implementation of practices in 
projects and functions within the OU must be understood 
and identified 

Possible Critical Factors include: 

¢ application domains (or lines of business) 

¢ geographical breadth 

¢ disciplines (e.g., systems, SW, or HW engineering) 

¢ effort types (e.g., development, maintenance, or services) 

¢ project types (e.g., legacy or new development) 

¢ customer types (e.g., commercial or government agency) 

¢ lifecycle models in use within the organization (e.g., spiral, 
evolutionary, waterfall, or incremental) 


e Sample project and support groups in the Organizational Scope 
must represent all critical factors identified and documented in 
quantifiable terms in the appraisal input and ADS 
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Organizational Scope Example? 


SCAMPI V1.2 Sample Description-1 
e Organizational Unit: Division XYZ, United Widgets 


e Projects excluded and rationale 
e All Research Projects — short duration and piloting new processes 
e Project XV11 — legacy using processes established Jan 2001 


e OU Size: 30 projects, 300 people 


e Critical Factors: 
e Application domains: Missile Command and Control 
Sonar signal processing 
MDA Information Technology 
¢ Geographical Breadth: Sunnyvale CA 
Denver CO 
Pittsburgh PA 
e Project Types: Development 
Sustainment 
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Organizational Scope Example? 


SCAMPI V1.2 Sample Description-2 


e Organizational Scope 
e Project ABC — 84 people, development (Missile C2), 2 sites 


(Sunnyvale, Denver), Focus 
e Project Alpha — 22 people, sustainment (Sonar),1 site (Denver), 


Non-focus, PA:... 
e Project Delta — 52 people, development (Sonar),1 site (Denver), 


Non-Focus, PA... 
¢ Project SEI — 42 people, sustainment (Missile C2), 1 site 


(Pittsburgh), Non-focus, PA:... 


e Organizational Unit Coverage: 
¢ Population: 66% 
e Project: 13% 
e Application domain: 66% 
¢ Geographical breadth: 100% 
e Project Type: 100% 
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Instantiation 


CMMI practices are abstractions which are implemented 
and made real in their application and implementation by 
projects and organizations. 


The context within which the practice is applied governs the 
implementation. 


The details of the implementation as well as the context 


within which the practice is implemented is referred to as 
the instantiation (organizational or project). 
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Focused Investigation 


An appraisal is conducted as follows: 

e A representative sample within the organizational unit is 
identified (i.e., projects and support groups). 

e The implementation of each model practice is reviewed 
for members of the sample set. 

¢ The extent to which each model practice is implemented 
across the sample set is representative of the extent to 
which the practice is implemented in the organizational 
unit. 
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Objective Evidence 


Documents or interview results used as indicators of the 
implementation or institutionalization of model practices. 
Sources of objective evidence can include instruments, 
presentations, documents, and interviews. 
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Two Types of Objective Evidence 


CMMI appraisal methods are based on the consideration of 
objective evidence as the basis for formulation of 
weaknesses, strengths, and ratings. 


SCAMPI Class A appraisal method only requires two types 
of objective evidence: documents and interviews. 


Documents include hardcopy, softcopy, and hyperlinks. 


Interviews are exploratory or focused questions targeted at 
managers, practitioners and/or users. 
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Organizing Objective Evidence 


In SCAMPI appraisals, objective evidence 
iS Organized using practice implementation 
indicators (PIIs). 


Plls are applicable to any practice or 
activity: 
¢ The conduct of an activity or the 
implementation of a practice will result 
in “footprints” which are attributable to 
the activity or practice. 
e They are necessary or incidental 
consequences of practice 
implementation. 


© 2006 by Carnegie Mellon University CMMI v1.2 Upgrade Training, Module 8: Appraisal Refresher - 082506 - Page 25 


—>—.. Carnegie Mellon 


= Software Engineering Institute CMMI. 
a 


Three Types of Plls 


Direct Artifacts are tangible outputs resulting directly from 
implementation of a practice (e.g., typical work products). 


Indirect Artifacts are artifacts that are a side-effect or are 
indicative of performing a practice (e.g., meeting minutes, 
logs, reports). 


Affirmations are oral or written statements confirming or 


Supporting implementation of the practice (e.g, interviews, 
questionnaires). 
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Example of a PII Characterization 


PP SP1.1: 
Establish a top-level work breakdown structure (WBS) to 


estimate the scope of the project. 


Direct artifacts: Indirect artifacts: 
- top-level WBS - meeting minutes 
- task descriptions - team charter 
- work package descriptions - WBS development notes 
Affirmations: 


-“| worked on the WBS team.” 
-“We used the WBS to generate the estimates.” 
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Developing Appraisal Results 


For SCAMPI B and C appraisals, practices are 
characterized based on objective evidence. These practice 
characterizations and strengths and weaknesses 
supporting these characterizations are the only appraisal 
outputs. 


For SCAMPI A appraisals, practice characterizations are 
developed at the project level, then aggregated to the 
organizational unit. Goal ratings, process area satisfaction 
and (optional) maturity level or capability level ratings are 
then determined based on these practice characterizations 
and associated weakness statements. Outputs include 
ratings, strengths, and weaknesses. 
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SCAMPI Appraisal Methods 
Fundamental Concepts 


Becoming a Lead Appraiser or Team Leader 
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Requirements for becoming a 
SCAMPI A Lead Appraiser 


Be sponsored by an SEI Partner. 


Participate as a team member on two SCAMPI A appraisals 
or one SCAMPI A and two SCAMPIB or C appraisals. 


Successfully complete SCAMPI Lead Appraiser Training. 


Successfully lead a SCAMPI A appraisal under the 
observation of a qualified observing lead appraiser (can 
also lead SCAMPIB or C appraisals). 
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Requirements for becoming a 
Certified High Maturity SCAMPI A 
Lead Appraiser 


Phase 1 — Effective Oct 2006 

¢ Submit an application describing education, training and 
experience as describes at 
http://www.sei.cmu.edu/certification/scampihmlia.html 

e Upon acceptance of the application, an oral examination 
will be scheduled by the SEI Appraisal Program 


Phase 2 — Effective Oct 2007 

e Pass a written test based on the High Maturity Body of 
Knowledge (in development) 

¢ Upon passing the written test, an oral examination will be 
scheduled by the SEI Appraisal Program 
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Requirements for becoming a 
SCAMPI B&C Team Leader 


Be sponsored by an SEI Partner. 


Participate as a team member on two SCAMPI A 
appraisals or one SCAMPI A and two SCAMPI B or C 
appraisals. 


Successfully complete SCAMPI B&C Team Leader 
Training. 


Successfully lead a SCAMPI B appraisal under the 
observation of a qualified observing lead appraiser. 
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Summary 

This refresher provides only a high-level summary to help 
you understand important appraisal concepts. 

For more information, see the SEI web site at: 


http://www.sei.cmu.edu/cmmi/ 
and 


http://www.sei.cmu.edu/collaborating/partners/ 
trans.partners.html. 
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Contents / Agenda 


New Affirmations on the Appraisal Disclosure Statement (ADS) 


SCAMPI Method Changes 
- Sampling Requirements 
SEI Quality Assurance Program 
Auditing of Every SCAMPI Appraisal 
Site Visits and Corrective Actions 
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Policy Changes 

Three Year Period of Validity 

- Required Upgrade Training and Certification 
Certification Program 


Application/Qualification Process 
Oral Examinations 
Phase II: Body of Knowledge (BOK) — based Certification 
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SEI Appraisal Program 


Program Responsibilities: 


Develop and maintain appraisal 
requirements and method 
descriptions 


Develop, maintain and teach 
appraisal courses 


Evaluate and maintain Lead 
Appraiser/Team Leader 
authorization, renewal and 
appraisal records 


Assure reliability and quality of 
appraisals for stakeholders of the 
CMMI Product Suite 
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New Features for Version 1.2 
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Value-Added Changes 


Strengthened Method Definition 
Appraisal Planning 
Appraisal Conduct 
Appraisal Reporting 


Changes Based on: 
V1.1 Appraisal Experience 
User Change Requests 


Sponsor Direction 


Steering Group Prioritization 
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Important Changes in Planning SCAMPI 


Refined definition of “Organizational Unit” 

Clarification of Soonsor and Lead Appraiser responsibilities 
Specification of “Focus Projects” and “Non-Focus Projects” 
Procedures for collecting data from other projects — if needed 
Identification of critical factors in sampling 

Definition of “Incremental Appraisal” 


Prohibition against “Delta Appraisal” 
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Important Changes in Conduct of SCAMPI 


Detailed specification of start and end criteria — with 90 day limit 
“One Appraisal, one Organizational Unit, One Rating” 
- No project or discipline-specific rating (unless project=OU) 


Documents used in appraisal must pre-date the start of the appraisal 


Critical sub-processes under Statistical 
management, and the process areas to 
which they relate must be documented in 
the finial findings. 


Strengthening peau a rel eyiTiayg 
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Important Changes in Reporting SCAMPI Results 


Appraisal Disclosure Statement (ADS) Greatly Expanded 
Documentation of OU and Sampling — in quantified terms 
Listing of critical sub-processes under statistical management 
Mapping the critical sub-processes to the organization’s objectives 
Expiration date for the appraisal results 


Affirmations from the Lead Appraiser and Sponsor 


= AStaevareldalcvallale povOvAU iad Med ecrelleyi ii avg 


== Software Engineering Institute | CarnegieMellon _ willHayes, November 15, 2006 


© 2006 Carnegie Mellon University 


Quality Assurance Program: 
CMMI Steward Responsibilities 
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Using Defined Processes 


Focus on Capturing “As-Is” Process 


- Forging a shared understanding 


- Clarifying roles and responsibilities _ a, i a a ésinmkatcs Helles 
- Looking for efficiencies : 
Quality Gate 1 - Inventory 
Quality Gate 2 - Verfication 
With a Common Understanding 2 Ha 
See 
- Evolution of the Process ; ie | 
De asec ae vs 
- Avoid “Process Replacement” i al 
Address Other A : f ‘ 
‘ Wo rk for M easu rab | e Pp rocess issue(s) Compliance Issue(s) Ethics Issue(s) 
Improvement | 
Wrap-up Quality Process 
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Paper Audits and Communications 


Active Audit Program Underway 
Unusual trends reviewed 


Follow-up with Lead Appraisers 
and with Organizations 


¢ Corrective actions taken 
We Have Prioritized Our Efforts 
Focus on areas of potential high 
risk to our stakeholders 
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Using historical data 
Working in partnership with SEI- 


Authorized professionals 
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Good News From Quality Assurance! 


SEl-Authorized Lead Appraisers Want Strong Quality 

Professionalism in our work is a shared value 

Lead Appraisers say “Thank You” when we audit them! 

Every participant in the process learns from these activities 

Good news from a QA audit is very nice to hear 

Bad news is being received very professionally and objectively 
Important Work is Usually Not Easy Work 

Management commitment at all levels (SEI & CMMI Steering Group) 

A shared value for assuring a common ethical standard exists 


Clarifying boundaries, and preventing problems will be the challenge 
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Policy Changes 


Immediate Adoption of V1.2 ADS 
No More Level for Life! 
HMLA Certification 
Sun-setting Policy 
31 August, 2007 
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Maturing the Profession of Process Improvement 


Though it is a Young Profession 
Many noteworthy accomplishments 
Substantial investment — globally 
Strong community of practice 
Many well-regarded people 
Standards of acceptable practice 
are moving to a greater level of 
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specificity 
Phased Approach 
Phase I: “High Maturity Lead 


Appraiser Certification” 
Phase Il: Formalized Body of 
Knowledge & Certification Exam 
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Status of Phase | Activities 
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Application Design 


Very Effective Community Survey Identified Important Pre-Requisites 


The Application Comprised of the Following Major Sections: 
High Maturity Appraisal Experience 
Education and Training 
Experience Designing and/or Implementing High Maturity Practices 
Experience Designing and/or Delivering High Maturity Training 


Near-Term Plans for Conducting High Maturity Appraisals 
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Application Scoring 


Established, reviewed and revised by certification team 
Piloted on selected sample of 10 applications scored by 4 independent people 


Initial Application Scoring Rules 
Refined based on statistical analysis of inter-rater differences 


Refined Application Scoring Rules 
Designed to minimize variability of interpretation 
Applied by at least two people scoring each application 
Cut-Score Established Using Pareto Analysis — A Clear ‘Dividing-line’ Emerged 


Every “Rejected” Applicant was then Invited to Re-Apply: 
More detailed set of questions tied to the scoring criteria sent in email 
A number of applicants were accepted based on re-application submission 
AStaevareldalciallale povOvAU iad MOdecrelleyi ii avg 
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Application Processing Data 
Application Scoring Data 
121 Applicants 
— 84 reviewed thus far 
— 34 to be reviewed 


84 Reviewed thus far 
— 66 passed 
— 18 failed 


18 Failed 
— 13 appealed 
o 9 passed upon appeal 


Scored Appeal 
mFailed Passed 
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Total 
To Be Reviewed 


@ Reviewed 
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Developing the Oral Examination 


Oral exam questions, and the “listen for” phrases 
Developed based on input from Expert Group 


- Reviewed among the Certification Team 

- Pilot tested with SEI internal and external staff 

— Expert as well as novice examinees included in beta testing 

— Involved large proportion of examiners in each session 
— Detailed discussion of scoring criteria conducted 


Conducted multiple “level-setting” sessions with examiners 
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Questions 


AStaevareldalciallale povOvAU iad MOdecrelleyi ii avg 


=>} Software Engineering Institute | CarnegieMellon _wittHayes, November 15, 2006 


lop-ulelMer-Taat-rel (mi (1 elameiicieti ag 


Ore) al esveimlaliolanitsiulela 


Will Hayes 
Quality Manager, SEI Appraisal Program 


SI O7-N Ved POLO) .\ m6 im AC) hs) =i O1\/ ON =i DLO) 


= Software Engineering Institute |§ Carnegie Mellon 


Redstone Arsenal, Alabama 


Office of the Chief Information Officer/G6 


The Road to Lean CMMI ina 
Small Federal Organization 


Susan Bassham 


Command Data/Applications Support 
(Business Area) 
Application Information Processes 
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e About Us 

¢ CMMI Status 

¢ Terminology 

¢ Construction Detours 


— Changes 
— Goals 
¢ Navigation 
— Road Map 
¢ Lean/Six Sigma 
— New Direction 
e Lean CMMI 


¢ Repairs Along the Bumpy Road 
e Current Points of Interest 
e Future Destination 
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About Us 


¢ Small Government Software Development 
Organization (70 government & 70 
contractor employees) 


¢ Develop business applications in support 
of aviation and missile mission 

¢ Software products consist of web based 
applications, support of AKO fielded 
websites, mainframe sustainment 
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CMMI Status 


¢ Currently assessed at SW-CMM Level 2 
¢ Implemented Lean CMMI activities 
¢ CMMI Appraisals Conducted 

—Class C 

— Class B 


Preparing for SCAMPI A Appraisal in early 
next year 
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Terminology 


¢ Value Stream Map (VSM) — standard method for 


documenting current and future states of 3 flows in total 
system: physical product flow, information flow, and time 
flow. 

¢ Kaizen — Japanese word meaning “take apart and make 
better,” a standardized approach to problem solving. 

¢ Lean — concept that focuses on improving customer 
value by removing the 8 deadly wastes: waiting, 
underutilized people, defects, extra processing, 
transportation, motion, inventory, over processing. 

¢ Lead Time (LT) — Total flow days from customer order 
to receipt of full order. 


¢ Cycle Time (CT) — hands-on touch time without wastes. 
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Construction Detours 


Barriers 


e Worked toward SW-CMM Level 3 


— Technical Working Groups 
e Started and stopped 
¢ Developed additional documentation 
— Piloted the Level 3 processes 
e Little buy-in 
e Duplication 
e Frustrated employees 
e Lengthy development cycle for large products 
— Average delivery 374 days 
— High defect rate 
— Process took too long 
e Attempted to define process for multiple project types 
— Urgent, small products 
— Larger products 
— COTS products 
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While we detoured... 


Things were changing 


Lean/Six Sigma 


e Army Materiel Command (AMC) embraced 
Lean/Six Sigma and directed adoption by 
Subordinate Commands 

e AMCOM Leadership required continuous 
improvement using Lean/Six Sigma Tools 

¢ Conducted Value Stream Map in Feb 2005 
— Identified current wastes, delays, duplication 
— Refocused teams on a long term ideal state 


— Developed a near term future state 
(chunking approach) 
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Why Change? 


¢ Software Engineering Process (SEP) needed 
improvements 
— Too complex 
— Duplication across documentation 
— Took too long to complete, especially for small products 
— Delayed product delivery 
e Sunset of the SW-CMM 
¢ Keep/increase customer base 
¢ Better utilize our resources 
e¢ Improve morale 
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Goals 


e Level the workload and share resources to accommodate 


increasing demands ees 
am 


¢ Provide first time quality 
¢ On time delivery aon 
e Reduce lead time 


e Standard, easy to follow Map 
— Streamline 
— Eliminate duplication 


¢ Collect valuable metrics — Visual at a Glance 

e Improve customer and employee satisfaction 

e Improve communication within the organization 
¢ Achieve CMMI maturity level 


11 
NDIA CMMI Conf-S.Bassham, Oct06 


Navigation with a Map 
and New Direction 


Customer , Nom-Value Value Added “Use, 
Added Activity Activity Inv Bp ow 
ee 13 


am : ) 
Policies ee Form App Output oc? Q “Ye, 
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‘ By Software Engineering Process 
(SEP) 


¢ Loop 0 — Concept Definitions and Product 
Requirements 


¢ Loop 1 — Job Planning and Analysis 
e Loop 2 — Design and Development 
¢ Loop 3 — Implementation and Test 
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Value Stream Map Event 
— Implementation Plan 
¢ Kaizens, Projects, Tasks 
— Insight into Lean Concepts 
e Reduce wastes 
¢ Eliminate delays and unnecessary approvals 
¢ Focus on flow 
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New CMMI Direction yg 


e Performed initial Gap Analysis with Class C 


e¢ Trained employees, supervisors, and 
champions on Intro to CMMI Version 1.1 


¢ Developed first cut at a Lean(er) SEP 
Created Milestone Schedule for future 
appraisals and process releases 

Conducted Class Bs on ML 2 Process Areas 
— Identified deficiencies 

— Held Kaizen events to address weaknes 
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Business Objectives 
(used for appraisal focus) 


¢ Develop lean CMMI software development 
capability to better respond to customer 


e Improve product quality 
e Improve process quality 
e Increase productivity 
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Repairs Along the 
Bumpy Road 


Lean Events 


e Conducted VSM 31 Jan — 4 Feb 05 


¢ Held 26 Lean Events during 05 and 06 


— Requirements and Planning, Workload Management, Design and 
Development, Web Standards, QA, CM, Metrics, Risk 
Management 


¢ Employees empowered to make decisions and quick 
changes 


¢ Briefed PCCB (formerly Known as SEPG) at end of each 
event 


— Make decisions real-time 
— Accept, reject, modify proposed process changes 
— Reduce delays in implementation 
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SEP Version Releases 


Incremental Implementations: 

e Jun 05, Version 5 (Interim) 

¢ Dec 05, Version 6 (Loop 1 Rollout) 
e Feb 06, Version 6.1 

¢ Feb 06, Version 6.2 

e Mar 06, Version 7 (Loop 2 Rollout) 
e Jul 06, Version 8 

¢ Aug 06, Version 9 

e Sep 06, Version 10 (Loop 3 Rollout) 
¢ Nov 06, Version 10.1 
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Improvements 


Changed Terminology 

— Process to Value Stream 

— Phase to Loop 

— SEPG to PCCB 

— Project to Product 

e Introduced Lean Thinking 
— Faster, incremental releases 
— Test driven approach 


— Quality built in from the start 
— Better teaming 
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‘Bik Supervisor’s Working Group 
eS (SWG) 


¢ Both Government and Contractor Reps 
¢ Develop Enterprise Plans for customers 


¢ Meet weekly to balance the workload 
and staff jobs 


¢ Record Lead Time status per job 
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Process Configuration Control 
Board (PCCB) 


¢ Membership from each role (Champion, SWG, TL, 
SPL, LAD, QE/Tester, CM/IA) 


¢ Communicate Value Stream Map changes to peers 
¢ Facilitate buy-in throughout the organization 


e Approve SEP Change Requests and manage SEP 
Version Releases 


e Assist in training the workforce 
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Changed Metrics 


374 days 105 days 

Phases 94 

# Activities 
6 


3 
0 (Collect data) 
#Approvals [65 
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Lessons Learned 


¢ People resist change — prepare for CAVE people 
e Focus — minimal task switching 

¢ Collect measurements often, check against goals 
e Visual status boards help manage workload 

¢ Pros/Cons to Interim Rollouts 


e Peer Groups — key to successful continuous 
improvement 


¢ Teaming with Wagon Wheel approach 
“Test-First” Approach reduces defects 
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Standardization: 


~ © NEN 


“SS” What's in it for me? 


e Makes decision-making easier! 

e Reduces errors, defects, rework, cost! 

e Improves quality and customer satisfaction! 
e Reduces frustration! 


e Is the baseline for improvement! (no 
Standardization, no systematic improvement!) 


Standardization is key to teamwork! 


27 
NDIA CMMI Conf-S.Bassham, Oct06 


Curren 
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Visual Management 


Reduce wastes of searching, defects, waiting: 
e Visual graphical flow for each Loop 

e Visual Online Help with drilldown 

e Step by step checklists by activity and role 


Loop 0 SEP VSM 


Legend CIO-G6 Software Engineering Process (SEP) Sim Product 


Team 


WW Team MTG 


% Loop 0: Concept Definition & Product Requirements 


Risk Mgt ‘oal is lays y BVCS. 7 : : = 
29 September 2006 : : aes = 
= LS : d lng | 
; ras i au duc 


CMIIA } ) 


) Production }# ; ; —| 
_ Scoreboard rk Pl 


3. Audit Schedule 


I> Opps to Improve —| 


1 


Decision Tree 8. Tot wane {-——_—— a ; 
Staff —nitiate or update s g : a OGUCUB ENR Management — eee 
a s ¥ u = ‘| ctivities 
Suspend z ¥ = - : ; 
Feasibility Study 
More Customer Info 
Reject 


cM 
Plan 


Metrics 


{ 


Exit Criteria: 
Set up product in - j i 2 
DOORS & PVCS (Di = Create\Praduct Finish product reqts aD Draft Final pecusee eeceer 
Notify PT and SLIM Schedule meeting Estimate in SLIM Negotiate job(s) size] [Exit Criteria: | ea ee one S| IP QACR Mantifving lie: 
personnel of product Conduct meeting Determine Negotiate timeframe] } Functional reqts Draft acceptance test Ga Sale iterhigil) a Rhee, 
name and id Capture data ——— Lifecycle Analyze/plan risks ‘Job reqts ‘cases Analyze/plan risks (mitigation) 
Set schedule for Exit Criteria: Prepare Top Level (mitigation) Performance reqts Baseline functional Exit Criteria: Link mitigation to requirement (create new 
customer meetings Voice of customer Design Link mitigation to User documentation requirements module in} | Acceptance cases requirement) 
Set expectations Scope / description Kam. Exit Criteria: requirement (create reqts DOORS Testable requirements Peer review — determine/review probability, 
Get customer Problem statement t e Top Level Design new reqts if Non-functional reqts Peer review Tech approach loss, and exposure for risks (metrics) : 
commitments Current environment = Prototype (if needed) necessary) Scope -Fix reqts Preliminary design a P' 
Prepare pre-QSADS (it Job expectations Risks Request timeframe Usecneranmance rents “Pass reqts Constraints/assumptions VENEERS (eaDICAUTSS I. 
new db or server space) Key date/delivery SLIM Product commitment for PT User rainingreats -Determine/review Pol | | Deliverables Track, manage, and control risks as 
Identify risks drivers Estimate If known: Security/privacy reqts rating for risks Milestone/schedule requirements 
Collect requirements Customerrelerences: Request additional Accessibility reqts -Validate reqts Adqd'l resources/cost As new risks and/or requirements are identified, 
and record in DOORS acronyms, derinitions; development staff Installation reqts Analyze risks Total cost estimates follow change management process 
Initiate Software Product and abbreviations 2 Special SW reats Revise and Finalize Defined, planned, Report/communicate risk scores as necessary 
Plan (SPP) Customer (a> Maintenance reqts SPP analyzed, and mitigated hi mlinelevatad 
Verify requirements within responsibilities y, Product/job(s) size, CM audit baseline Cls | | risks/reqts when angiieleval 
PT ; Risks ~ ince ‘ 2 Software Product Plan 
Begin collecting product Product Team effort Define Risk and Mitigation r ‘\ Audit Findings Report 
effort data in SLIM data ¢ Select “Move to Loop 1” in 
S DOORS Functional 
Reqts. Module to trigger 
end of Loop CM and QA. 
activities. 
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Output pe 


Risk Mgt 


Risk 31 
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Overarching Activities 


Product Demo 
Ensure test 
readiness 
Required participants: 
- SPL 
- LAD 
- Dev 
- Customer 
Invite as optional: 
TL 
- QE/Tester 
- CM/IA 


Exit Criteria 
Validated Scripts 
Test Readiness 


Provide 
Acceptance 
Cases to 
Customer 

Train Testers 
(Cust.) if 
necessary 
Record & Rework 
defects; Retest 
Cust. Acceptance 
(in SYN/Chg) 


2 
=” 


CIO G6 Software Engineering Process (SEP) 


Loop 3: Implementation and Test 


Goal is 30 days 


Ss 
S 
ot 


29 September 2006 


=) 
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Exit Criteria 


Customer Tested 
& Staged Product 
ready for Training 
Accept in 
SYNERGY/Change 
Audit Findings 
Report 

Tests 

- Acceptance 
Cases 

- Security 

- Integration 


Schedule 
training 
Conduct 
Training 


Exit Criteria 
Users 
Trained & 
Feedback in 
Info. System 
ASEC 
ATONATO 


- Record URL in AKO 
- Report into APMS 


Re-compile 
Publish Final 
User Doc 
Notify WW that 
product moved 
to production 


—==> 


Exit Criteria 
Available 
Working 
Approved 
Product 


8. Causal Analysis 
7. Trend Analysis 
6. Metrics 


5. Lessons Learned 
4. Audit Findings 

3. Audit Schedule 

2. Process Deviations 
1. Opps to Improve 


Risk Management Process: LE 
ra y 

Identifying Risks ' (—7 
Analyze/plan risks (mitigation) = 
Link mitigation to requirement (create new 
requirement) 
Peer review — determine/review probability, 
loss, and exposure for risks (metrics) 
Validate requirements/risks 
Track, manage, and control risks as 
requirements 
As new risks and/or requirements are 
identified, follow change management process 
Report/communicate risk scores as necessary 
when and if elevated 


Store Code 
Update data (In 


QACR 


SPP) 

Collect, Analyze, 

Report: 

-- Customer 
Satisfaction 

-- LT 

-- Quality 

-- OTD 

Baseline the 

product 

Lessons Learned 


End Job = 
rc) 
= 


Exit Criteria 
Satisfied Customer 
who accepted job, 
reported satisfaction 
to give CDAS a 
Completed, Measured, 
Reported, Auditable 
Job 
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Lean Internal Training 


¢ Continuously improve 
— Add, Change, Modify, Delete 


— Innovative deliveries and celebrations 
e Rollout Celebrations 
¢ SEP Fair 


¢ Provide high level overview 

e Specialized Peer Group training sessions 
— Hands on tools 
— Simulate! 
— Role expertise 
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Future Steps 
on 
Our Lean Journey 


Visual Status “at a Glance” 


e Visual Management using metrics data 
— Enables better decision making 
— Insight into opportunities to improve 
— Allows monitoring of lead time and other 
metrics 
e Dashboard concept 
— Data at your fingertips 
— Information for all roles 
— Meet the needs of managers and data calls 
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“4h Continuous Improvement! 


¢ Utilize Lean/Six Sigma Tools 
— Apply creative problem solving 
— Put SEP controls in place 
— Measure results 
e Lean CMMI 
— Enhance business/management processes 
— Standardize 


— Improve Customer Satisfaction (product and SEP) 
— Plan/prep for Level 3-4 appraisals 
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Questions? 
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Contact Information 


Susan T. Bassham 
susan.bassham@redstone.army.mil 
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Glossary 


PCCB - A collection of specialists from each role including management who facilitate 
the definition, maintenance, and improvement of the process(es) used by the 
organization. 


SWG - An authorized group within the organization that balances the workload across 
the organization; provides product tracking and oversight; and assists in the 
implementation, sustainment and continuous improvement of the Software 
Engineering Process. 


TL - Leadership role on the Product Team who is responsible for Loop 0 (Concept 
Definition & Product Requirements) and ensuring the success of software product 
development. 

SPL - Technical leadership role on the Product and Development Team responsible 
for success of software product development. 

LAD - Leadership role on the Product and Development Team who is responsible for 
design and development of the software product. 


QE/Tester — Technical role on the Product and Development Team who is 
responsible for Quality Assurance and Testing activities during the software product 
development. 

CMIIA — Technical role on the Product Team who is responsible for Configuration 
Management and Information Assurance activities during the software product 
development. 
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Customer Success Is Our Mission 


Easing project planning for 
Small Programs 


Ken Weinberg 
El Segundo, CA 
kiweinberg@raytheon.com 


November 15, 2006 


Raytheon 
Agenda 


e Small and Large Program Characteristics 

e Comparison of Three Processes for Large and Small Projects 
e Desired Single Process for Large and Small Projects 

e History of Small Project Process 

e Improved Process with Guidance for Small Projects 

e Sample Small Project Templates 


These thoughts are based on my experiences working 
with projects for over 20 years, and do not necessarily 
reflect experiences across Raytheon 
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Raytheon 
Typical Small Program Characteristics 


e Staffing for 3-8 Engineers 

e Program Life 12 Months 

e Small Proposal Activity 

e Limited Customer Participation 

e Single Product Deliverable/No Data Deliverables 
e Minimal Customer process focus 

e Limited Requirements 

e Standard Facility and Resource Requirements 
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Typical Medium/Large Program Raytheon 
Characteristics 


e Staffing for 100 Engineers 

e Program Life 5-10 Years 

e Significant Proposal Activity 

e Regular Customer Participation 

e Significant Contract/Data Deliverables 

e Customer Process Expectation 

¢ Defined Requirements 

¢ Defined Methods of Verifying Requirements 
¢ Good Profit Opportunity 

e Unique Facility and Resource Requirements 


Standards (CMMI, ISO) Written for Large Programs with Typical 
Program Phases 
Organizational Processes Derived From These Standards 
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Org Process Designed for Medium/Large = ®aytheon 
Programs 


e Extensive Planning Phase 

e Involve Stakeholders 

e Extensive Schedule with Dependencies 
e Program Managed with Metrics 

e Formal Requirements Traceability 

e Extensive Testing/test Levels 

e Significant Management Interest 


e Formal Communication Important to Keep Project Teams 
Coordinated 


Good Candidates for Appraisals 
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Raytheon 
Small Project Process Needs 


e Limited Planning Phase 

e Involves Fewer Stakeholders 

e Schedule with Major Milestones 

e Metrics Used to Convey Program Status to Management 

e Derived Requirements with Limited Traceability to Higher Documents 
e Creative Methods of Verifying Requirements 

e Limited Testing/test Levels 

e Less Extensive Management Interest 

e Formal Communication is a Burden 


Small Projects Can Follow Good Process, but ... 
¢-Do Not Need as Much Formal Communication Among Team Members 
eCannot Easily Afford to Produce Enough Artifacts to Make Good 
Candidates for Appraisals 
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Raytheon 
The Challenge 


e Standards (CMM, CMMI, ISO, Corporate Initiatives) written 
for large programs 


e Organization processes derived from these standards 


¢ Small projects can follow good process, but 


— A large formal process may be a burden 
— Often find “process” intimidating 
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Comparison of Three Processes for 
Large and Small Projects 


e Planning Phase 
e Metrics 


e Testing/Requirements Verification 


Raytheon 
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Raytheon 
Planning Phase Characteristics 


¢ Medium/Large Program 
— Extensive planning phase 
— Detailed IMP, IMS, staffing plan, interdependencies 
— Customer imposed requirement specification 
— Facility Plan 
¢ Small Project 
Brief planning phase 
Schedule with major technical milestones, staffing plan 
High level requirements (or goals) 
Facility planning 
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Planning Phase - Similarities and Raytheon 
Differences 


e Similar 
— Planning Phase 
— Schedule 
— Statement of Work 
— Budget 
— Staffing Plan 
— Requirements 
— Facility Planning 
e Different 
— Details 
— Interpretation 
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Raytheon 
Metrics Characteristics 


e Medium/Large Program 
— Extensive Metrics (Technical, Programmatic, Quantitative) 
— Used to Monitor, Manage and Improve Program 
— Convey Status to Management and Customer Monthly 

e Small Project 


— Limited Technical and Programmatic Metrics 


— Scoped Version of Standard Metrics used to Monitor and Manage 
Program 


— Convey Status to Management 
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Raytheon 
Metrics — Similarities and Differences 


e Similar 
— All programs track progress with metrics 
— All programs report metrics to management 
¢ Consistent reporting format 
¢ Compare trends across organization 
e Different 
— Details and frequency 
— Usage 
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Testing/Requirements Verification Raytheon 
Characteristics 


¢ Medium/Large Program 
— Formal peer reviews 


Multi Level Independent Reviews with Engineering Technical Experts and Management, 
and Customer 


Extensive/multi-level testing 
All requirements verified 
Formal documentation/records 
— Customer participation 
¢ Small Project 
— Informal peer reviews 


— Combined single Independent Review with Engineering Technical Expert and 
Management 


— Single level testing 

— All requirements verified 

— Informal documentation/records 
— Often no customer participation 
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Testing/Requirements Verification — 
Similarities and Differences 


e Similar 
— All programs conduct peer reviews 
— All programs conduct independent reviews 
— All programs verify requirements 
e Different 
— Details 


Raytheon 


Page 14 


Common Process Themes From All Raytheon 
Examples 


e All Types of Programs Benefit From Process Discipline 


e All Types of Programs Follow Core Process 
— Planning 
— Requirements 
— Metrics 
— Testing/Verification 
— Configuration Management 
e Different 
— Scope/Details 
— Interpretation 
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Raytheon 
The Goal 


e Use the Same process for large and small projects 
— Compliant With the CMMI Model, ISO/AS9100, Corporate Standards 
¢ Keep Directives Short and Simple 
— Provides project buy-in to process 
— Generic wording 
Document Requirements in SRS & Document Requirements 
SOW =» Tasks 
— Almost no required formats 
Smaller projects rely heavily on supplemental non-directive guidelines 
and templates for tailoring and implementation guidance 
Pre-Tailor directives not used by most small projects (e.g. Formal 
Customer Reviews) 
— Scoped from full process to cover characteristics used by most small projects 
— Review scoped tailoring for further refinement 
May add back in items tailored out when appropriate 
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Raytheon 
History 


e Large Process 
— Fully compliant with CMMI models 
— Produced artifacts to make assessments/appraisals easier for appraisal teams 
— Used model “jargon” 
— Overwhelming for small projects 
e Initial Small Software Process — Based on Products: Requirements 
Document, Test Plan, Version Description Document, etc. 
— Used only portions of directives related to products 
e Used large process - unclear which portions applied 
e Non-uniform process — not applied consistently 
e Not conducive to process improvement 
¢ Not compliant with standards 
e Separate Software Small Process — Scoped Specified Directives Into 
New Directive System 
— Used existing infrastructure support 
— Achieved over a 75% reduction in directives, pages and paragraphs 
— Still a separate directive system 


Improved Single Process with implementation 
guidance for smaller projects 


Page 17 


Raytheon 
Full Process 


Directive/ High Level 
Non-Tailorable Directly Traceable 
to CMMI, ISO, Corp Stds 


Lower Level, 
Directive/ Further Direction on 
Tatlorable “How” to 


Meet Requirement 


Guidelines/ 


Non-Directive 
Templates 
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Improved Process 


. Enablers 


| Multi-Part Enablers 
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Raytheon 
Tailoring Template Example 


—_ P= 


Small projects only need one costing procedure (see Par. 5.10a). Cost realism analysis (Par. 5.11) not 

El-01-16 Cost Estimating x required. All bids need mgt approval; projects <$1 require Dept. Mgr approval, projects >$1M 

require Center Mgr approval 
El-01-21 Process Tailoring ee. le ea al 


El-10-02 Project Measurement & Analysis 


Review 
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Raytheon 
Sample Tailoring Template Detail 


DOC TAILORING Comments 
Number CODE 


El-14-04 roe Design Oeene aan and ete ACCEPT See small project template 
Readiness) Management Review 
E|-14-05 | Test Readiness Management Review Nenrednice ia projects designated Technology 
Demonstration 


El-14-08 | Independent Review of Start-up Plans ACCEPT Small Project Templates Used for plans 


El-15-01 | Peer Review CHANGE If effort is upgrade of existing system, only changes and 
————s 5 interfaces need be reviewed. 


ACCEPT This need not be a formal document with a specific format. 

E|-02-01 | Software Requirements DOORs or an EXCEL spreadsheet are OK as long as 

they are controlled after baseline. 
EI-03-05 | Software Preliminary Design ACCEPT Preliminary and Detailed Design Phases are often 
=e es combined on small projects 

Software Test Plan and Software Test Description may be 

E|-04-01 | Software Integration and Testing CHANGE combined, or included in another document. Software 

Test Report may be red-lined Software Test Description 


Plan for regressing test is re-running selected sections of 
El-04-13 | Software Unit Test CHANGE Software Test Description, if analysis shows regression 
‘aoa ee testing is required. Unit testing (and unit test 
documentation) may be combined with integration 


El-04-14 Software Corrective and Preventative ACCEPT This is generally accomplished during monthly project 
se Action leadership meetings. | Page 21 


Work Product Management Plan Raytheon 
Example Template 


Work Product Control Level Checklist/ 
Stakeholder Involvement Note: Unless otherwise noted, all items are stored on the project server 


Date: December 12, 2005 Stakeholder Roles/Disciplines 

Control Level Function | Peer Program] Line org SE [SW |Config/ 
Responsi | Rev ||Customer |Manager|Center/Dept}IPTL |IPTL|Data Mgt] Quality 

es ar eal swf | 


PARIS |S 


[Software Problem Reports (SPR) | OT CO eT Sw AR 


4 
Sw QuaityPlan—SSSSSCid id x Pd RS 
Sweode CSCS id x i wr 


[SW Version Description Document | | |x | « |swem] |_| | [A JAR] 
SwoMReponsSSSSCCSC~d Cid Tr dT TT CCT CCC 
SW Coding Standards_—————+t| | xf} | sw_|_| |_| | _JARy 
SwomPlan SSCS Sid xd 
SwPeerRevews SSS SY dw AR 


Sofware Requirement Documentation | | |x|] sw |x || | A_Jar|s___]r__ 
Sofware Design Documentation | | x ||] sw_|x_]| |_| || aR |[s___|r__ 
[Sofware Test Documentation | | |x |] sw_x_| |_| |’ _AR|S__|r__ 
Sofware TestRepot_———SSCSCSCSr Sid x rd RS 


When Identifying Stakeholders use the following codes 

R = Responsible for producing the completed work product 

A = Responsible for approving the work product 

S = Provides support in the production of the work product 

C = May be consulted in the generation of the work product 

| = Needs to be informed of the completion of the work product or any changes to the work product 

F = Can be used to facilitate the development of the work product 

N = Not applicable to the development of the work product. This code may be used to provide clarity w hen several stakeholders are involved in the generation of a work product 


Ss 
Ss 
Ss 
A,R 
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Raytheon 
Summary 


¢ Goals 
— 1SO/AS9100, Corporate Standards, CMMI model compliant, as scoped 
— Smaller projects not planned to major role in appraisals 


e Method 
— Start with full process 
— Use generic wording where possible 
— Keep it short and simple = really short and simple 
— Scope for smaller projects 
— Rely heavily on non-directive templates and guidelines 
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Questions ? ? ? 


a= any a 


: en 


Getting to Level 3 Risk 
Management (#3772) 


= SElIPartner 


© 2006 Natural SPI 


Topics 


Comparison of Level 2 and Level 3 Risk Management , 1. 
Step by Step Approach to Level 3 ) 

The Desired Outcome 

References 


Questions 
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Getting to Level 3 Risk 
Management 


Comparison of Level 2 and | 
Level 3 Risk Management — 
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Getting to Level 3 Risk 
Management 


Level 2 Risk Management is a Snap 


Level 2 Risk Management consists of: 


O One Project Planning Specific Practice (PP SP2.2) — 
Identify Project Risks 


OQ One Project Monitoring & Control Specific Practice 
(PMC SP1.3) — Monitor Project Risks 


O Three pages of text 
Because this is a Level 2 activity: 


QO The process used does not need to be defined and 
can be different for each project 
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Getting to Level 3 Risk 
Management 
Level 3 Risk Management (RSKM) is an 
entire Process Area 


Level 3 Risk Management consists of: 


OQ Three Specific Goals with a total of 7 Specific 
Practices 


O Twenty-two pages of text (more than the entire 
Project Monitoring & Control section) 


Because this is a Level 3 activity: 


O Adefined process tailored from the organization’s 
process is required 


Level 3 Risk Management requires a comprehensive 
risk management strategy 
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Getting to Level 3 Risk 
Management 


RSKM Specific Goals and Practices 


SG1__—sCOwPPreparration for risk management is conducted 
SP 1.1 Determine risk sources and categories 


SP 1.2 Define the parameters used to analyze and categorize 
risks, and the parameters used to control the risk 
management effort 


SP 1.3 Establish and maintain the strategy to be used for risk 
management 


None of this was required for Level 2 risk management! 
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Getting to Level 3 Risk 
Management 


\ » RSKM Specific Goals and Practices (cont) 


SG2__ Risks are identified and analyzed to determine 
their relative importance 


SP 2.1 — Identify and document the risks 


SP 2.2 Evaluate and categorize each identified risk using the 
defined risk categories and parameters, and determine 
its relative priority 


Except for having defined categories and parameters 
this is also required in Level 2 (PP SP2.2) 
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Getting to Level 3 Risk 
Management 


\ » RSKM Specific Goals and Practices (cont) 


SG3__ Risks are handled and mitigated, when 
appropriate, to reduce adverse impacts on 
achieving objectives 

SP 3.1 Develop a risk mitigation plan for the most important 
risks to the project, as defined by the risk management 
strategy 


SP 3.2 Monitor the status of each risk periodically and 
implement the risk mitigation plan as appropriate 


Level 2 only required the risk monitoring aspects of 
this goal! 
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Getting to Level 3 Risk 
Management 


Risk management Is not % 
the same as worrying about 


your project!* 


* Waltzing with Bears by DeMarco and Lister, p. 64 


© 2006 Natural SPI 


Getting to Level 3 Risk 
Management 


Is Level 3 Risk Management Worth It? 


Level 3 Risk Management has many benefits: 


ee ee a 


Helping new project teams perform adequate risk 
management 


Improving budget and schedule adherence 
Predicting an adequate risk reserve 
Managing risk rather than watching risks happen 


Identification of systemic organization issues that can 
be resolved by management 


Understanding the actual impact of risk on your 
projects 


Learning through shared, relevant experiences 
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Getting to Level 3 Risk 
Management 


Step by Step 
Implementation 
Approach 


p 


Getting to Level 3 Risk 
Management 


QO Locate as much risk information from past projects | 
as possible. If you are a Level 2 organization, you 
should have risk lists and monitoring records. 


Where to Start 


Q If you are not a Level 2 organization, and don’t 
have risk lists and monitoring records, it’s 
probably better to wait until you do. 

Q Why start with historic data? 

Q If you don’t develop your risk management 
strategy based on actual experience, the end 
product may not be used because it is viewed 
as irrelevant. 

Q Select a relevant risk taxonomy for comparison 
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Getting to Level 3 Risk 
Management 


About Risk Sources and Categories 


Risk sources describe potential places where risk 
Originates 


Risk categories are buckets for grouping and 
analyzing risks so they can be dealt with effectively 


Risk sources and categories can facilitate thorough 
risk identification on all projects 


Thorough risk identification provides greater benefit 
to the project 
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Getting to Level 3 Risk 
Management 


Step #1 - Identify Risk Sources 


QO Risk sources describe potential places where risks 
originate. They can be: 


QO Internal to the project (for instance, the technology 
being used, a fixed schedule, complexity) 


O External to the project (for instance, weather, 
changing regulations, political instability of a 
country) 


O Review each risk to identify its likely source(s) 


QO After you’ve identified common sources of risk from 
your data, look at other taxonomies to improve the 
thoroughness of your source list 


LE 
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Getting to Level 3 Risk 
Management 


How Am | Going to Do This, Exactly? 


One way to facilitate this process: 


QO Print historic risk information (risk statement, impact, 
mitigation plans, etc) on index cards or other similar 
sized physical media — you may need to select a 
sample subset if too much data exists 


Use different stickers to identify potential sources 
Log the source for each sticker type 
Place stickers on the index cards as appropriate 


One index card can have multiple stickers! 


EP Eee ae 


Review the list of sources for consolidation 
Opportunities 


OQ Get more than one person involved! 
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Getting to Level 3 Risk 
Management 


Step #2 — Identify Risk Categories 


Risk categories are buckets for grouping and ™ 


analyzing risks so they can be dealt with effectively. 
Often categories are just a higher grouping of 
sources from a taxonomy, but how else can 
categories be used? 


QO Phase of the life cycle in which the risk usually 
manifests 


Q Organizational group responsible for handling the risk 
(Senior Management, Systems Engineering, Project 
Management, etc) 


Internal or External to the organization 


O What groups make sense in your organization? 
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Getting to Level 3 Risk 
Management 


Step #3 - Define Risk Identification Methods 


QO Balance thoroughness with creative thinking by 


combining a structured method with brainstorming 


Q Involve appropriate stakeholders, not just the 
project team 


O Structured methods can include: 


Walking through the WBS 
Walking through the requirements 
Walking through the project plan 


Walking through a list of risk sources or risk 
taxonomy 


Reviewing historic risk data and lessons learned 
from similar projects 
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Getting to Level 3 Risk 
Management 


Step #4 — Adopt Risk Statement Standards 


QO People often confuse issues, problems, and risks 


Q Arisk is an event or condition that may occur in 
the future, and which will have an impact on the 
project if it did occur 


QO Anissue is a risk that has already occurred, 
nothing can be done to prevent it 


QO A problem might have no project impact 


QO A standard risk statement format helps everyone 
identify and describe risks consistently: 


QO If {condition present} then {impact} 
QO If {event} then {consequence} 
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Getting to Level 3 Risk 
Management 


Step #5 - Clarify Impact and Probability Definitions 


Q Tailor impact definitions to your organization based 
on historic data: 


QO Impact includes more than just schedule and budget, 
it could include safety, quality, security, employee 
morale, or customer satisfaction 


O Safety issues could cause minor injury to one 
individual or serious injury to a large number of 
individuals 


QO Security issues can range from disclosure of 
individual personal data to issues of national security 


. \ QO Probability can be described using quantitative or 
qualitative values: 


O In either case, define and agree upon the meaning of 
these values in your organization 
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Getting to Level 3 Risk 
Management 


Step #6 - Define Standard Triggers and Thresholds 


OQ What standard thresholds are important? 
O Are some risk categories more critical than others? 


QO Do projects with a higher (or lower) than average 
level of risk require a different approach? 


Q Will projects with certain levels of risk simply be 
dismissed? 


QO What is the priority, impact or probability cutoff for 
risks that must consider mitigation? 


I Q What standard triggers are important? 


OQ Does the occurrence of a high priority risk trigger 
involvement of certain levels of management? 


Q Does the identification of a new, high priority risk 
trigger specific action? 
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Getting to Level 3 Risk 
Management 


Step #7 - Define Risk Responses 


There are many risk responses besides mitigation . | 
and contingency, including transferring, avoiding, 1. 


monitoring and possibly even ignoring! What risk 
responses are valid in your organization? 


When is each risk response appropriate? Provide 
guidance, not just options 


Which risks will have responses developed? Which 
will just be monitored? Which will be ignored? 


Define the expected activities associated with each 
risk response (for instance, accept versus ignore) 


Review historic data to identify risk responses that 
have worked in the past 
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Getting to Level 3 Risk 
Management 


Step #8 - Define Cost-Benefit Calculation Method 


~Q Every risk does NOT need a mitigation plan! 
_ QU Sometimes mitigation is just too expensive. 


Consider: 
QO The effort required to implement the mitigation 


QO The likely reduction in probability and impact of the 
risk as a result of the mitigation 


O Weigh the total cost of mitigating against the 
projected impact of the risk should it occur 


; . QO Establish a minimum threshold for when mitigation 
should even be considered, for instance, only risks 
with at least Medium probability and Medium Impact 
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Getting to Level 3 Risk 
Management 


| Step # 9 - Document the Risk Management Strategy 


Ay A Risk Management Strategy : | 
e ; ~~ O Is acomprehensive risk management approach for a 
project tailored from an organizational standard 


QO Translates organizational risk management 
expectations into a project-specific strategy, 
including: 

U Project-specific risk sources, categories, tools, 
measures, thresholds 

Q Risk tolerance of the project sponsors 

QO Overall approach to dealing with risk on the project 


O Can be part of a risk management plan and a subset 
of the overall project plan which needs to be 
reviewed with the project stakeholders 
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Getting to Level 3 Risk 
Management 


Step #10 - Identify Performance Measures 


Q What do you need to know about risk management? © 


QO Are we spending too much or too little effort on risk 
management? 


How well are we estimating probabilities and impact? 


How many problems occurred during the project that 
could have been anticipated as risks? 


How well are we characterizing overall project risk? 
How effective are our risk responses? 


How effective is our risk reserve allocation and 
usage? 


QO Will these measures enable you to monitor and 
control the risk management process? 
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Getting to Level 3 Risk 
Management 


Step #11 - Develop and Deliver Training 


=! Tailor your training to the various stakeholders in the 
process: 


O Customers need to understand their involvement and 
what risk related information they will be seeing 


O Senior Management needs to know how to support the 
process and what their role is 


O Project Managers need in-depth training on all aspects 
of the process 


QO Analysts, Designers, Developers, Engineers, Testers, 
Support Personnel need the ability to recognize and 
report risks to the appropriate individuals 
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Getting to Level 3 Risk 
Management 


Step #12 -— Perform Quality Assurance 


Implement process and product quality assurance: y 


_ Q Develop a checklist for verifying the risk management 
process is followed 


S 4 Q Identify risk related work products (such as risk lists), 
a determine which should be audited and develop 
standards to evaluate them against 


0 Incorporate these activities into your process and product 
quality assurance plans 
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Getting to Level 3 Risk 
Management 


Step #13 -— Address Generic Goal 2 


Some Level 2 Generic Practices may already be | . | 
addressed for Risk Management via Project Planning 1. 


_and Project Monitoring and Control, but consider: 


O Do your policies include expectations specific to risk 
management? (GP2.1) 


QO Are there new risk management planning tasks that 
need to be scheduled and assigned? (GP2.2, GP2.4) 


O Are new tools and job aids being introduced? (GP2.3) 


QO Are there new risk work products that projects need to 
maintain under configuration management? (GP2.6) 


QO Is the expected involvement of all stakeholders in the 
risk management process defined (including Senior 
Management)? Is there a way to track and monitor their 
involvement? (GP2.7, GP2.10) 
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Getting to Level 3 Risk 
Management 


Step #14 - Address Generic Goal 3 


To address Generic Goal 3, you should: 


O Document the risk management process and provide 
tailoring guidance 


QO Provide a mechanism to collect the outputs of each 
project’s risk management process for future reference 


O Provide a mechanism for individuals and teams to 
provide feedback and improvement suggestions based 
on their usage of the risk management process 
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Getting to Level 3 Risk 
Management 


is 


2 4 Implementation Looks 
and Feels Like 


What RSKM 


Getting to Level 3 Risk 
Management 


Typical RSKM Activities and Behaviors 


In an organization that has implemented CMMI-based " 
RSKM processes, you could see these activities and 
behaviors: 


ile 


ee 


Senior Management reviews the risks before 
approving project plans 


Risk reserves are allocated and monitored 
All team members can identify risks 
Risk identification is encouraged and rewarded 


Serious consideration is given to analyzing risks, 
developing and monitoring risk responses 
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Getting to Level 3 Risk 
Management 


Typical RSKM Work Products and Artifacts 


In an organization that has implemented CMMI-based 
RSKM processes, you will typically see these work 
products and artifacts: 


1. Risk Registers (Lists) 
Risk Strategies 


Risk Status Reports 
Risk Measures 


Ce eS. 


Risk Source and Category Lists 
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Getting to Level 3 Risk 
Management 


<_ The following references may be useful: | y ; 


1. A Guide to the Project Management Body Of 
Knowledge (PMBOK®) Third Edition, Project 
Management Institute, for more information visit 
www.pmi.org 


Risk References 


2. Continuous Risk Management Guidebook, Software 


Engineering Institute, for more information, see 
www.sei.cmu.edu/publications/books/other-books/crm.guidebk.html 


3. Waltzing with Bears, Managing Risk on Software 
Projects, Tom DeMarco and Timothy Lister 


The companion paper for this presentation includes 
additional references and is on the Conference CD 
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Getting to Level 3 Risk 
Management 


Contacts and more information 


QO Susan Byrnes, PMP 
Natural SPI, Inc 
e-mail: susan@naturalspi.com 


QO Natural SPI’s web site: www.naturalspi.com 


=> $ElPartner 


Real Process 
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Getting to Level 3 Risk 
Management 
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BAE SYSTEMS 


Interfaced vs. Integrated Process 


BAE Systems 

Electronics and Integrated Solutions Operating Group 
Nashua, New Hampshire 

USA 


Agenda BAE SYSTEMS 


—Organization Overview 
—Process Improvement Approach 
—Resulting Process Architecture 
—Process Improvement Results 
—Conclusions 


Electronics & Integrated Solutions BAE SYSTEMS 
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Sensor Systems BAE SYSTEMS 
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Austin Manassas 


at 


Merrimac | Nashua Lexington Washington, D.C. 


Process Improvement Time Line 


BAE SYSTEMS 


BAE SYSTEMS Lifecycle 


Management 


CMMI® Released by SEI 


PMBOK® Focus Introduced 


| 


Software Process 
Assessed at SW CMM® 
Level 4 (9/01) Previous 
appraisals at Level 3 
(94, 98, 00) 


System Engineering 
Process at EIA-731 L2+ 
going for L3 


Other Engineering 
Processes well defined 
& followed 


Program Management 
Process Defined, 
Inconsistently Followed 


2001 


IPDP Effort Begins 


5 Opportunities for 
Improvement (OFI) 
assigned to IPDP 


IPDP Identifies 
Solutions to 14 
Program Operating 
Model (POM) Issues for 
Leadership Ratification 


Update SW process to 
SW-CMM Level 5 


System Engineering 
Process Assessed at 
EIA-731 L3 (10/02) 


2002 


Customer Requirements for 
CMMI® Level appearing in 
RFPs 


Start new Projects 
using IPDP 


Full CMMI 
SE/SWI/IPPD/SS based 
Assessment of Process 
at Level 4 via Class B 
appraisal (10/04) 


IPDP Management Process 
Pilots March-June 


Implement IPDP Technical 
Processes for System 
Engineering and Software 
Engineering 


Deploy IPDP V1.0 (9/03) 


All Key Projects have 
transition Plans by 12/03 


Form Enterprise Process 
Group (EPG) 7/03 


2003 2004 


CMM and CMMI are registered trademarks of Carnegie Mellon University 


PMBOK is a registered trademark of the Program Management Institute 


External 
Drivers 


Implement Corrective 
actions from Class B 
appraisal 


CMMI SE/SW Class A 
appraisal at Level 3 
(4/05) 


Implement process 
improvements to 
enhance usability 


Expand Level 4 and 5 
behaviors from 
Software to Systems 


2005 


Background BAE SYSTEMS 


—Current Process Improvement Activities began in 2002 designed to 
—Address project performance issues identified in 2001 
—Take the capability of the process to the next level 
—Leverage the best practices in place 
—Within the local organization 
—Within sister organizations in BAE Systems 
—Basis for process improvement 
—Capability Maturity Model Integrated (CMMI SE/SW/IPPD/SS v1.1) and 


—The tenets of the Program Management Institute; Program Management 
Body of Knowledge 


Approach Overview BAE SYSTEMS 


—Process improvement activity to be organized as a project 
—Defined budget 
—Formally planned 
—Representation from all disciplines 
-Engineering, Project Management and support disciplines 
—Use a full system engineering approach to developing the process 
—Requirements analysis and definition 
—System Design 
—Implementation 
—Integration 
—Pilot (verification) 
—Deployment 
—Early decision needed on how to document the enhanced process 
—Knit existing processes together into an INTERFACED process or 
—Develop a single INTEGRATED process 


Types of Process Implementations (slide 1 of 2) 


Non-Integrated Process 


Ca 
5 OS 


Interfaced Process 


e Each function defines its own process to its own e Add defined Interfaces and interactions 
understanding of needs into each functional process (touch points) 
e Each function does similar things in different ways e Each function defines its own process to a 
a , ; . common understanding of needs 
¢ Communication, sharing of product and information 
is along narrow defined paths e Each function does similar things in 
¢ Collaboration is adhoc to non-existent compatible ways 


¢ Communication, sharing and collaboration 
are built into each functional process set. 


Types of Process Implementations (slide 2 of 2) 


Integrated Process 


Common 


Process 


¢ Common process for all functions with extensions as necessary 

e Each function does similar things in the same way 

¢ Communications, sharing and collaboration are a natural outcome of working to the 
same process 


Considerations 


BAE SYSTEMS 


Interfaced Process 


¢ Pros 


Less overall work 

Discipline processes to be executed 
are documented all together 

Level of detail of process can be 
adapted to the maturity of a discipline 
Infrastructure to support the 
processes Is already in place 


* Cons 


No clear owner of integrated 
behavior 


Total volume of process is greater 
More cost and effort to maintain 


Difficult to keep all parts of the 
process in alignment 


Integrated Process 


¢ Pros 


Single clear owner of integrated 
behavior associated with the process 
Easier to maintain 

¢ Single place to make changes 

¢ Smaller overall process description 
Less infrastructure require to Support 
the process 
Forms the basis of a more extensible 
process architecture 


¢ Cons 


Requires more work to develop and 
deploy 


— Requires more stakeholder 


involvement in reviewing and 
agreeing to the process 


Selecting the Approach: 


Analyze Pre-Integration Program Plan Hierarchy 


Engineering Plans 


«Systems (SEMP) 

*SW (SDP) 

*HW (HDP) 

*CM 

*-DM 

*Supportability 

Quality Plan *Test Equipment (ITEP) 


Program Plan 


Subordinate Plans 


*Operation/Manufacturing Process Plan 
*Material Plan 

‘Installation Support Plan 
*-Program-specific Capital Equipment Plan 
*-Program-specific Facilities Plan 

*DTUPC Plan 


Selecting the Approach: BAE SYSTEMS 
Analysis of Plans 


Planning Elements CMMI | LCM | MDP| Project | System | Software | PSEP CM Quality | HW| Support DM 
Project Scope 


—Plans were structured as stand-alone Seciomaae sila ae a 


Purpose/Mission/Objective's x x x x x x 
Benefits and Enablers x x x 

documents ost 
Success Criteria x x 
Vision x x x x x 
Customer Requirements and Assumptions x x 


— Not hierarchical conse z a : 


— Many common attributes Sass Come a Ng : 


Technical Baseline 


. Product Content _ : x 

—_ N O C O Nn S iste Nn cy Decision Analysis and Resolution (Trade Studies) x x : 
- = oer x x Oo 

—Plans were functionally oriented Fe log — = < a: 


Ff 
yas 


Project Unique Process x oe 


Process Appraisal & Audit Process x 
. [Schedule 
os Some lans contained rocess Project Schedule and GP Analysis cL x 
p p Integrated Master Schedule x x x x SS 


Major Milestones x x 


descriptions that were duplicated from a= soeee 


WBS Matrix x 


WBS Dictionary x 


project to project a ra 
O 


x 
x 
x 


Labor Budget by WBS over Life of Project 
Supplier and Material Dollar Budgets 
a Earned Value Techniques 
—Temp lates for p lans existed Gite Ded Cost Daler Suga Dp 
Rationalization of Budgets to Contract Dollars 
Labor Budget by CAM (work packages) 


Dollar Expenditure Plan Over Life of Contract 


— Checklists were not value added or eam 


Roles and Responsibilities 


I t t Interfaces/Coordination 
non-exisS en Resources 
Key Resources 
Facilities 
Staffing 


— Content not consistent across projectS — tysssrcomgy 


GFE/GFS 7 x 


x 
x 


x 
x 
x 
x 

x |< }< }< |x 


. . . Reuse 
or within projects a 
Capital Equipment 
Verification Equipment 
Stakeholders 
Related IRAD 
Stakeholder Involvement/Commitments 
Between Process Phases 
Conflict/issue Resolution 
Supplier Agreements 
Product Assurance 
Information Management 
Data Item Submittal/Management 


x |< }>< ] >< J >< J >< 


mi (oda gle) alecwrelacMm\V/(0)aom @xelanlaalelamumat-lamelalie(elcy 
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Moving forward with Requirements Analysis and Architecture Design BAE SYSTEMS 


Interfaced Integrated 


Integrated 


Common 
Process 


— Our analysis of Interfaced vs. Integrated set our direction toward an integrated process 


— Our analysis of existing project plans and processes served as an early validation that 
an Integrated process was possible and the correct approach 


—We needed to establish a process architecture that supported an integrated approach 


— Common procedure level descriptions 
— With natural, transparent extensions to support discipline needs 


A Good Process Architecture is the Key to a Successful 


Integrated Process 


Requirements Analysis Approach 


— Use the CMMI subpractices as discrete requirements 
— Add requirements from other governing process frameworks not included in CMMI 
—|ISO 9001-2000 (AS9100) 
— Program Management Institute Body of Knowledge (PMBOk) 
—BAE Systems Life Cycle Management 
— Develop specific requirements to describe how projects will operate 
— Called Program Operating Models (POMs) 
— Reviewed and agreed to by all functions and all levels of management 
- Approved by organization leadership 
— Examples: 
- Structure projects as Integrated Product Teams (IPTs) 


- How to staff and de-staff projects 
- Expansion of Configuration and Data Management into Information Management 


Single Database Created to Capture All Requirements 


Integrated Product Development Process (IPDP) Structure 


Common 
activities 
used by all 


All planning 
and oversight Wrelarele(=vaatoialt 
responsibility 
(including 


technical 
planning) 


Develops and Oferta r4cliteyarel aKs\evalaliers| 
maintains 


process All technical 
activities 


Requirements were Allocated to One of Four Zones 


Each Zone has Process Elements BAE SYSTEMS 


Management Zone 


Manager 
Corrective Assigned 
Action 


Charter Estimate Project Coy ali ce) ar. Close 
or ontrol Project o Project or 


Project Element Project Element Project Element 


Plan Project or 


: Project Element 
Requirements 


Manage Risk & 
Opportunity 


Requirements Grouped into Process Elements 


Anatomy of a Process Element 


BAE SYSTEMS 


Practices 
—One or more per process element 
—Contains 
— Inputs & outputs 
— Entry & exit criteria 
— Flow diagram 
— Steps to be performed 


Review Lists 
— At least one for each work 
product referenced by the 
practices 
— May be discipline specific 
— Contain rules for the 
development and review of the 
work product 
-Required rules 
-Expected rules 
- Project specific rules 


Templates 
— For work products that require 
specific formatting or 
— To provide the user witha 
starting point 


Practices, Review Lists and Templates Form the 
Process Building Blocks 


Overall IPDP Architecture 


Overarching documents, such as 
BAE Systems Policies & Life Cycle Management 


Required 


Practices 
Review Lists 
(some templates) 


j1e19q JO |aAaN7 Hulsess9u| 


WK=Jin) ele-icct-wm a tclalelolere) Com me-lialiare| 
FVaye| 
@)daYslmCrel(erslalercm\ytclaclatel 


Value of this Architecture BAE SYSTEMS 


—Practices are written at a high level 
—Common to all disciplines 
— Describe what has to be done 
— Describe how to the level of consistency and commonality needed 
— Overall process is work product centered 
— Each work product has a review list 
- Describes the elements that must exist in a quality, compliant work product 
— Allows each discipline to have unique review lists to support their needs 
- Example: 
- Common Size Effort Review List 
- Size Effort Review List - Software 
- Size Effort Review List — Mechanical 
- Size Effort Review List — Electrical 
- etC... 
— Work products can be 
- Used to scale (tailor) the implemented process. For example: 
- A table style work product can define the frequency of meetings and events 
- A table style work product can define the level of stakeholder involvement 
- Shared and reused between different documents on the project 
- Used to keep larger documents living (up to date) 


Examples of Planning Work Products 


BAE SYSTEMS 


¢eTemplates and review 
lists ensure consistency 
and facilitate development 
of each planning element 


e Review lists at each level 
of planning hierarchy 
define required elements 
and content 


t— Description 

t— Objectives/Requirements 
t- Shared Vision 

t— Key Success Criteria 

~~ Enablers 

L- Cost and Schedule Targets 


-_ Assumptions 


t~ Major Milestones and Events 


IPT Structure 

Roles and 
Responsibilities 
Operating Procedures 


t- WBS 
__ Dictionary 


Control and 
Approval Matrix 


L- Roles and Responsibilities 
t~ Involvement Plan 

tL- Commitments 

“ Communications 


t~ Material Plan 

L- Supplier SOWs 

t— Supplier Management 
Plan 

‘_ Supplier Agreements 


t— Integrated Schedule 
— IMS/IMP 
_ Critical Path Analysis 


t—- Lifecycle 

t- Defined Process 
“Monitor and Control 
Reviews 


Labor 
Material and 
Suppliers 
ODC 


t- Staffing Plan 

L- Knowledge, Skills and 
Training 

t- Facilities 

t- Equipment/Tools 

_ Capital Equipment 


t~— Product 
Structure 

L- IV&V 
Approach 

-_ Make/Buy 
Trades 


Progress 
Quality and 
Process 
Technical 


— 


meee Control Board ae 


Risk and Opportunity 
Parameters 

Risk and Opportunity 
Review Board 
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Examples of Practices, Review Lists 


and Templates 


. Partition requirements into logical sets or groups to 
cilitate analvsisvand desion nf the nrnchy 


Requirements Review List 


orndiaict 


The requirerient shall be identified by a single ) Definition 


sentence that contains the word “shall” or “must”, 


Definition 
Definition 


The requiretnent shall be unambignons, 


The requirement shall be clear 


The requirement shall contain acomplete Completeness 


Derre and develop technical gp gaumcmetbe 
requirements based on the operational analysis, 


existing technical solution, and the verification report 
from early validation activities. 
Requirements Review fist 


interjace Requiremenis Review List 


Plan 


BAE SYSTEMS Inforimation and Flectrome 


Requirements Specification Review: List 


e Systems 
ev— <date= 


Allocate requirements and constraints to people, 


requires 
published 
Specification 


This section will be performed when the publi 
NX document format. 
Collect developed materials and create document(s). 
Concept of Operations Review List 
penations Template 
interface Specification Review List 
Interface Specification Template 


shing of developed materials requires 


Rule Type | Ble S| Problem Type 
NJ 
The Requirements Specification shall include any 


Required The Requirements Specification shall include all Completeness 
. 
control number assigned. 


requirements’ for the product or product element. 
ait 3 Sj ¢ The Requirements Specification shall reference any | Completeness 
—_—T published documents that constrain or further 


Completeness 


Perform Peer Review on the document(s). 


Practice lays out the steps 


specific to a work product 


Template aids in formatting 


Review List provides the rules and are 


Peer Review 


BAE SYSTEMS 
P Pemptate tame Requirements Specification Template 
= — Sai arte : 


<Project Name> 


<Product or Product Element Name> 


Requirements Specification 
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BAE SYSTEMS 


Integrated Product Development Process 


A Single Integrated Process BAE SYSTEMS 


Engineering IPDP 


Program Management 


Operations 


Integrates the Major Functions on Development Projects 


IPDP is: 


—A full implementation of the CMMI SE/SW/IPPD/SS v1.1 model 
— Includes practices through Level 5 
— Adds definition and implementation to Lifecycle Management framework 
— Incorporates best practices from 
— Other BAE Systems business units 
— Industry sources 
— Program Management Institute Body of Knowledge (PMBOk) 
— Addresses identified project performance issues 
— Supports different project types and customer communities 
—A fully integrated process definition 
— One approach used at multiple levels for 
- Management — planning and monitor & control 
-Requirements, Design, Integration and Test 
— The process was appraised using a Class B appraisal in October ’04 up to and including 
Level 4 
— Project corrective actions from that appraisal were closed by February ’05 
— Organization corrective actions were closed by December ’05 
—CMMI SE/SW model used for successful Class A appraisal at Level 3 in April 05 
— Process included IPPD aspects but process areas and goals were not assessed 


Pre-IPDP Discipline Practices by Process Areas BAE SYSTEMS 


- Zz i i) ~~ 
- zm = 7 _ 
- i - 7m z= 
- i = 7 i 

c) 60% 

a) 

O 

W”) 50% 

s) 

c 

ao) 40% 

— 

2 
- i» 7 zz Zz 
- i zz a _ 
- i z= a Zz 


0% 
Systems Software Hardware Supportability 
This is a breakout of Practices only. 


Guidebooks, templates and review 
@ Management ® Organization G Universal B Technical lists are not included. 
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IPDP Notional Breakout of Technical Processes BAE SYSTEMS 


Notional Scale 


Conclusions BAE SYSTEMS 


—We created an Integrated process that is 
— Easy to maintain 
— Supports multiple project types and customer communities 
— Has been successfully deployed 
—A highly extensible, well documented process architecture 
— Allows new processes to be added in a simple easy way 
— Resists changes that create a Plate of Spaghetti process 
—A comprehensive process requirements database 
— Documents how each process requirement is implemented 
— Makes it easy to understand the impact of any purposed change to the process 
— Makes it easy to understand the impact of any change to the requirements 
— Lessons learned 
— Heavy stakeholder involvement is needed early and often in the process development 
activity 
— Maintaining buy-in by the disciplines without all the attributes of ownership is difficult 
— All requirements need to be captured, not just unique requirements 


Final Thoughts 


¢ Questions 
«Comments 


Contact information BAE SYSTEMS 


Robert Rice 

Process Planning and Architecture Lead 
Enterprise Process Group 

Performance Excellence 


BAE Systems Electronics & Integrated Solutions 


Tel: 1 (603) 885-3917 


email: robert.e.rice@baesystems.com 


Customer Success Is Our Mission 


Staying Atop Mt 
CMMI 


Thomas Lienhard 
Raytheon Missile Systems 


Timothy Davis 
Retired — Raytheon Missile Systems 
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Customer Success Is Our Mission 


The Never Ending Story .,.,.,. 


situation normal 


Obtain Level 


a 


Gather 
more rocks 


Appraisal event ——, 


Spin story —/ 

Front runner 
Programs 

told to engage 


CMMI Level 


Mandate 
Scramble to 


identify gaps 


\ 


Gather/polish 
more rocks 


—— Appraisal 
event 


~~ Rocks 
gathered 


1st Qtr 24 Qtr 3'¢ Qtr 4 Qtr 


Common story seen over and over again 


Customer Success Is Our Mission 


Breaking the Cycle 


This has been played out over and over 


Some organizations who have been through this before, do 
put things in place as not to back-slide 
— Reward the programs and people 
— Capital Investments 
— Establish new goal 


But most still back slide, Why? 


ee 


= 


Raytheon 


Customer Success Is Our Mission 


Sustaining Foundational Pillars 


¢ Sustaining High maturity is dependent on how solid is the 
lower foundation 
— Management Commitment and Support 
— Resources 
— Program Buy-in 
— People 
— Process 
— Organizational Objectives/Goals 


— Infrastructure 


Is Yours a House Built on Sand or Bedrock? 


Raytheon 
Spreading the Support Across Pillars’”” 


¢ Success starts with Management Support and Resources 


¢ As the other pillars are established, take hold, and become 
part of the culture, they become less critical 


If the infrastructure is sound, 


sustainment is possible 


Raytheon 


Customer Success Is Our Mission 


Foundation Attackers 


High Maturity is not a perpetual machine. It takes effort to sustain the gains or 
the foundation will eroded and the structure will collapse 


— Lack of continued sponsorship 
¢ Organizational changes 
¢ Got the Level - attention is elsewhere (acquisitions, crisis of the day) 
— Competition for Resources 
¢ Funding directed to other initiatives 
Program’s were never bought in, it was a mandate 
¢ Program schedule more important than process improvement promises 
No career path for people, process not seen as important 
¢ Staff turn over - Shift in personnel and teams’ makeup 
Process is too cumbersome, one size fits all, no proven ROI 
¢ Process for process sake 
— The Number was the Goal 
¢ Deploy narrow and high not wide 
— Lack of accountability 


Complacency and Atrophy Take Over 


Raytheon 


Customer Success Is Our Mission 


How to Mitigate the Attackers 


Be aware they exist 

Understand the journey, not destination 
Plan and allocate for future years 

Get the pillars built, deployed, and improve 
Treat Process Improvement like a program 


Plan, Monitor, Verify, Report, Improve 
— Resources, training, stakeholders, risks 


Raytheon 


Customer Success Is Our Mission 


Proven Insurance 


¢ Must begin with a strong foundation 
¢ Management must be committed, active, INVOLVED sponsor providing adequate 
resources 
¢ Must be a career path to attract and retain people with right skills 
— Availability is not a skill 
¢ The number cannot be the goal 
¢ Can’t rest on your laurels — an organization is always moving — it may be forward or 
back 
¢ Process that works for the particular business and objectives 
— Not one that was “bought” 
— Must show benefit or ROI 
— Process for process sake 
¢ Programs understand why they would want to do this 
— Not just - your bonus depends on it 
— Balance short term schedule success with long term improvement promises 
¢ Make sure there is a goal of maintaining current maturity 


— Goal posts mark the end zone, Without them in sight we may not realize we are getting farther 
away instead of closer. 


— On-going periodic reviews and surveillance audits 
¢ Establish and maintain dedicated teams (measurement, EPG, Quality Improvement) 


Use the Infrastructure to Fight Off Attackers 


Observations That Appear to woe 


¢ Use an evolutionary approach providing pre-planned 
increments of capability 
¢ Upgrade the related support infrastructure 
¢ Continue involvement in external activities which are 
shaping CMMI 
¢ Mature existing processes instead of creating new 
processes 
¢ Allow programs to mature as well as the organization 
— Programs include what is important to them 
— Programs allowed to innovate and the organization replicates 


¢ Cover the entire lifecycle: Concept, Development, I&T, 
Manufacturing, Product Shelf Life 


Must Add Value to Programs, 1.e., ‘The Business 


Top 10 Excuses NOTto S2vimeon 
Sustain High Maturity 


. ’'m too busy, it’s chaotic around here. 
. We’ve got it under control. 

. Pll lose my creativity. 

. We’ll just buy a tool! 

. We'll work smarter. 

. Not important to my boss. 

. My organization is different. 

. We’re going to reorganize. 


1 
2 
3 
4 
5 
6 
7 
8 
9 


. We have too much “real” work to do. 
10. We don’t do process. 


Raytheon 
P ro b [ e mM S E Nn CO U N te red Customer Success Is Our Mission 


¢ Some levels of management were uncomfortable with the true 
capability 

¢ Aggregating on-going programs with completed programs 

¢ Rationale Subgroup assumptions were incorrect 

¢ Treating all measures as normally distributed 

¢ Statistically controlling an indicator (SPI, CPI) rather than a 
subprocess (design defect removal rate) 

¢ “Process Dodgers” 


¢ Statistical “Purists” 


Process Physics — Some Things Just Take Time 


Raytheon 


Customer Success Is Our Mission 


Qu eS tl ONS hmniniilee ents Cag ideation 


Contact: 


Thomas G Lienhard@Raytheon.com 
Timothy_J_Davis@Will!Work4U.com 


Missile Systems 


y Motivating the Team 

to Continuously 

Improve after the 
CMMI Rating 


Raytheon Missile Systems 
Lety Santillan and Bill Borkowski 


November 16, 2006 
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Author Contact Information 


Leticia C. Santillan 
Enterprise Process Group 
Raytheon Missile Systems 
Tucson, AZ 


Missile Systems 


William J. Borkowski Jr. 
Engineering Fellow 
Raytheon Missile Systems 
Tucson, AZ 


Leticia C Santillan@raytheon.com Wjborkowski@raytheon.com 


920.794.2338 


920.794.2444 
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Missile Systems 


How Do You Start Motivating?? 


It is Critical that You Create a Supportive Environment 
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How do you Create an Atmosphere that » ae 
Supports Improvement? issile Systems 


» Focused vision and goal 

=» People aligned to common goal 

= Voice to the “Top Decision Makers” 
=» Co-Location 

» Open communication 


» Open environment where critical issues 
can be discussed 


= No fear 

a Okay to fail 

a Okay to be a unique individual 
= Have fun!! 


Environment, Communication, and Focus are Key 
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Missile Systems 


Our Success Story 


» Our team worked hard to achieve 
CMMI® goal 


a Strong leadership commitment 
=» Lots of communication 

= Daily meetings 

=» Supportive environment 

» We had a lot of fun 


When You are Working as a Team Anything is Possible! 
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Missile Systems 


Our Moment of Courage: We lost our way 


=» Leadership changes happened at 
the top level 


=» Management moved on to new 
positions 


» Many team members went on to 
new programs 


=» There was a new process of the 
day but our organization was not 
in charge of it. 


=» No goal in site 
» No focus 
» Little direction or vision 


How Do You Find Your Way Once You Lose It? 
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Missile Systems 


How do you get it back? 


» Used 6 Hats by Edward De Bono 
s Had Fun 


=» Leveraged our past improvement 
towards new initiatives 


» New Communication Plan 
=» Leadership was invigorated 


» Common Agenda was to improve our 
product and processes 


» New focus 
»s Communications 


Creating New Process for Every New Initiative Costs Too Much 
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Our Success Stories 


» Program was a frontrunner for our 
process improvement activities. 

» Once the goal was achieved program 
manager continued strong 
communication. 

» Common focus on producing best 
product. 

» Accountability was factor. 

» Leveraged existing process as new 
initiatives/requirements were given. 

=» Customer like the approach and 
awarded more work (new Contract). 


Missile Systems 


When We Leverage Off What Already Exists We Do It Better 
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Tips for Maintaining or Regaining Raytheon 
Momentum Missile Systems 
» Communication 

» Enterprise wide vision and voice 


» Leveraging of existing process 
and infrastructure 


a Strong communication plan and 
execution 


» Accountability at all levels to ensure 
successful deployment 


It is Easier to Maintain or Salvage than to Start from Scratch 
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Customer Success Is Our Mission 


Questions 2 


Thank you for your participation 


Raytheon — 
Enterprise Level of Process 
Improvement 
National Defense Industrial Association 


6th Annual CMMI® Technology Conference 


and User Group 
November 13-16, 2006 


Linda A. Kovar 


Author’s Name(s): 
Nancy M. Raymond 
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: Raytheon 
Introduction 


e This presentation explains the Space and Airborne Systems 
(SAS) Opportunity Improvement Board (OIB) and its process 


for 
— Collecting process improvement proposals 
=" i, — Identifying innovations for the organization that will help its 
v quality and process performance 
— Prioritizing and analyzing the proposals with respect to the 
e& organization's objectives 


— Selecting proposals for further consideration 


— Determining the cost and return on investment of the 
proposal 


— Piloting the implemented proposals to select which ones to 
deploy to the organization 


— Deploying improvements that provide positive impact to the 
organization's objectives 


— Measuring the effects of the deployed improvements 
Copyright © Raytheon Company 2006 as unpublished material. All rights reserved. 
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Benefits of Enterprise Level Raytheon 
Improvements 


e The desire to help programs and the 
organization be successful drives 
organization improvement 


Improvements are implemented to 


— Help programs meet their quality and 
process-performance objectives 


— Improve the organization’s performance 
on current and future programs 
A CMMI level 5 organization’s focus Is: 


—“... on continually improving the range of process performance through 
both incremental and innovative improvements.” (Chrissis, p. 77) 


Help programs succeed with improvements that are 


aligned with the organization’s goals and objectives. 
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Identifying Improvement Raytheon 
Opportunities 


e Anyone in the organization can identify an enterprise 
improvement opportunity through GuBmnIEFeeebzeR) 
— Process change requests 
— Data analysis (program level and organization level) 

— Symposiums and conferences (internal and external) 
— Lessons learned (programs and organization) 
— Other feedback to the organization 


e Feedback is collected and tracked on-line as part of the SAS 
Enterprise Management System (EMS) 


5S | 
/SAS PROCESSE 


ENTERPRISE MANAGEMENT SYSTEM 


Process & Technology Improvement Ideas System 


Copyright © Raytheon Company 2006 as unpublished material. All rights reserved. 
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Raytheon 
Prioritizing and Ranking 


e The enterprise established an Opportunity Improvement 
Board (OIB) to prioritize and rank the submitted 
improvement opportunities with relationship to 


— The organization’s goals and objections 


— The statistical understanding of the organization’s current process 
capability 


— The estimated cost-to-benefit ratio 
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Raytheon 
Stakeholder Involvement 


e The Opportunity Improvement Board (OIB) membership 
includes key personnel from the Enterprise Process Group 
— EPG Lead is the Chair 
— All disciplines represented 
— All sites represented 


SPACE AND AIRBORNE SYSTEMS 
ENTERPRISE PROCESS GROUP 


e The OIB recommendations are reviewed and approved by 
the Integrated Process Steering Team (IPST) whose 
membership includes senior representation from 
— All engineering centers 
— Program management 
— Supply Chain Management 
— Quality 
— The EPG 
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High Level Opportunity 
Improvement Flow 


Submitted 
Process & 


~metrics analysis 
—lessons learned 


~ deployment feedback 
— process improvements 


- trip reports 


~benchmark results 
—~conference feedback 


~R6o6® results 


~ appraisal results 


Technology 
Improvements 


Opportunity 
Improvement 
Board 


Steering Team 
(IPST) 


Programs 
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Raytheon 


Prioritizes and ranks 


Reviews and Approves 


Executes Improvement 


-_Define La 
-Pilot m 
-Measure R6 
-Deploy O 


Raytheon Six Sigma 


All rights reserved. 
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EPG IPT Activity | EPG, Mgmt, Tech 


__OIB Activity epg activity 


Detailed 
Improvement Flow 


Annual Process 
$< Improvement planning 


GS eee 


Collect 
Metrics 


Deploy per EMS 
deployment plan 


No — reject 
& document 


IPST Review Analyze 


R6o0® Results 


Pilot 
Develo 
o | p 
. Improvement 
No — reject R60 Concept 
Raytheon Six Sigma Brie f 
& document 
Yes 


Develop 
Process? 


Create BOE 


No — reject 
& document 


Prioritization 


EPG IPT Activity 


OIB Activity 


EPG Activity 


EPG, Mgmt, Tech 


Details 


Prioritize improvements 
and identify innovations 


Analyze cost & benefit of Proposals 


No — reject 
& document 


cost: benefit ratio 
sufficient ? 


Improves Org’s 
Processes? 


IPST review 


Prioritization 
Matrix 


QFD 


No — reject 
& document 


Prioritize 


and identify improvements 


innovations 


IPST Review 


Prioritization 
Matrix 


QFD 
Select for 
further 
consideration? 
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Improvements are prioritized Raytheon 
based on weighted criteria 


Supports Business Strategy (Goals/Metrics) 
Scope of Improvement 
(Expected percentage of programs helped) 


Objective Data to Substantiate Need for Change 
Probability of Successful Implementation 


Proposal is Supported (by organization metrics) 


Expected Return on Investment (ROI) 
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Priority improvements are Raytheon 
mapped to Goals (QFD) 


WHAT vs. HOW 


Symbo 


Streagth of Relationship 


eo) oo] Wale 


_ 


Title: | 245 EPG O1B 2005 Selection 


Date: 


E 
o 
& 
mo 
o 
o 
ao 
a 
a 
irJ 
oa 
wa 
P=] 
_ 
a 
= 
2 
= 
a 
= 
= 
= 


Develop ARO Template 
Cleanroom Process 
PMP Engineering Merge 


Appraisal Tool 
ME EM Process 


Im portasce 


Test Verticality Methodology 


| 1 [Cham Levels subprocesses | 5 | OH TT TTT 
Ae EG 


| femme sintw TTT 

ES aa (ee (ee (ae (el (ER (Se 

Be Pos Alignment Tt 

| & [Maintain tO Compliney | TTP 
SSS 


whats 


Techaical Targets 


— =a 
Competitor Benchmark $$ $f f+ — 
— es ee ee ee eee ee ee es ee ee ee 
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Implementation 
Details 


OIB Activity 


EPG IPT Activity | EPG, Mgmt, Tech 


No 


PN 
R6o 


Raytheon Six Sigma 


Develop 
Improvement 


Pilot 
Improvement 
Concept 
Brief 


Create BOE 


Develop 
Process? 


EPG Activity 


No — reject 


Annual Process _ 
Improvement planning 


Deploy per EMS Collect 
deployment plan Metrics 


Yes No — reject 


& document 
Deploy? 
, Deploy? = — SS 


R6o® Results 7 
Pilot 


—<—$—<——_ 


Develop 
Improvement 
Concept 
Brief 


Yes 


Develop 
Process? 3 

No — reject 
& document 


Create BOE 


BOEs include: 
eBarriers/RISkS 
«Cost 
«Schedule 
eExpected ROI 


& document 
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Raytheon 
Concept Brief 


e The Concept Brief describes the proposed implementation 
including: 
— The benefits to the programs and organization 
— The impact to the practitioners 
— The transition and deployment plans 
— Implementation details 
— Schedule 
— Resources 


e Concept briefs are reviewed with the stakeholders prior to 
implementing the improvement 
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Raytheon 
EPG Oversight 


e The Enterprise Process Group (EPG) 
— Implements the selected improvement projects 
— Monitors the status of the improvement projects 


— Manages the deployment of the implemented improvement projects that 
are chosen for full scale deployment 


— Measures the impact of process improvements on the organization’s 
capability 
— Manages the OIB process throughout the year 


— Reviews opportunity improvement suggestions as part of its annual 
planning 


SPACE AND AIRBORNE SYSTEMS 
ENTERPRISE PROCESS GROUP 
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Measurement 


Raytheon 


e Improvements are measured collectively 


e The organization’s process capability is measured and analyzed quarterly 


to understand the impact of the collective improvements 


Expected Range of Values _0.50 to 1.44 


X =0.97, s=0.16 
February 2005 n= 133, Cy, = -0.02 


Process Capability for CPI 


LSL = 0.98, Nominal = 1.0, USL = 1.1 
40 


frequency 
frequency 


10 


0 
0.25 05 0.75 1, d.25 1:5 1.75 


CPI 


Expected Range of Values 


December 2005 


Process Capability for CPI 


0.62 to 1.33 
X =0.98 s=0.12 


n=185 C,, = -0.05 


LSL = 0.995, Nominal = 1.0 


0.25 0.5 0.75 


1 
CPI 


1.25 


1.5 


Process Improvement shown in Program Cost 


Nov 2004 - Mar 2005 


¢ Supply Based Optimization 
¢ Objective Evaluations 

e¢ Program Planning 

e EMS Process Tailoring 

e Peer Review 

¢ Defect Logger 

e Integrated Mgmt Review 

¢ Work Product Mgmt 

¢ Technical Solution Data 

¢ Statistical Process Control 
e Process Assets Library 

e HW Development Planning 
¢ Managing with Metrics 


0.86 T T T T T T T T 


Mar 2005 - May 2005 


e Structured Decision 
Making 

e Lessons Learned 

¢ Risk and Opportunity 


and Schedule Performance 


Management 
e SW Integration & Test 
¢ FRACAS 
1.16 5 
1.14 4 
1.12 4 
11 4 
1.08 + 
1.06 + 
1.04 4 
1.02 | 
Ou: 
Obvae | 
0.96 4 
0.94 4 
0.92 4 
nat ] * 115 4 
0.86 0.88 0.9 0.92 0.94 0.96 0.98 1 1.02 1.04 1.06 108 11 1 ‘ae | 
SPI 107 4 
1.05 4 
1.03 + 
1.01 
O 099 + 
0.97 + 
0.95 
0.93 + 
0.91 + 
0.89 
se 0.87 + 
0.85 


0.86 0.88 0.9 0.92 0.94 0.96 0.98 1 102 104 106 108 11 112 114 116 
SPI 


May 2005 - Dec 2005 


e Interface Management Process 
¢ Qualification by Similarity 

¢ Update Gap Tailoring Tool 

¢ Cost Collection 


¢ FRACAS 
e PLE 
¢ RMS 


¢ Mechanical Engineering 
¢ — High Reliability 
e Systems Safety — Phase | 


* COTS 
e Updated Lessons Learned 
Database 
12/05 @ 
@ 


0.86 0.88 0.9 0.92 0.94 0.96 0.98 1 
*Major ECP authorized last year, not yet reflected in EVM 


02 1.04 1.06 108 11 1.12 1.14 1.16 
baseline. 


Ops 


Raytheon 
Summary 


e The SAS organization improvement process 
— Designed to help the programs succeed 
— Supports the Organization’s goals and objectives 
— Involves appropriate stakeholders 
— Results in improvement to the organization’s process capability 


@. 


at 
“= 
wa 
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Questions 
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Raytheon 
Contact Information 


e Linda A. Kovar 310.334.1828 
lkovar@raytheon.com 


e Nancy M. Raymond 972.344.5451 
n-raymond@raytheon.com 
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Biography 


e Linda Kovar is a Senior Manager of Programs at Raytheon Space and Airborne 
Systems in El Segundo, California. She is the Enterprise Process Group (EPG) 
lead for their process improvement activities. In August 2005, she successfully 
led the activities in achieving a combined Systems, Software and Hardware CMMI 
Level 3 for the business unit which encompasses 7000 engineers in several 
states. She is currently leading the Enterprise Process Integration activities to 
focus on a CMMI Level 5 maturity across all of SAS Engineering. Linda has 25 
years with Raytheon where she has held positions in Program Management, 
Functional Management, Technical Staff, Test Program development and 
software engineering. She is a certified Raytheon Six Sigma expert, has a BS 
degree in Computer Engineering from the University of California Los Angeles 
and is a graduate of the Anderson School of Executive Management. 


e Nancy Raymond is the Deployment Support Cross Product Team Lead in the 
Space and Airborne Systems Enterprise Process Group. Nancy has been with 
the company for 28 years. For most of her time at Raytheon, Nancy has beena 
software and process engineer on programs and within the organization. Nancy’s 
specialties include software planning, estimation, and metrics. Nancy has been 
involved with the development and deployment of CMM/CMMI compliant 
processes for more than 10 years. Nancy has a Masters Degree in Mathematics 
from Carnegie-Mellon University. 
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REGISTRATION 


CMMI and ISO 9001 


A Registrar’s Perspective 
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e Explore Fundamental Differences 
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¢ Choosing a Criteria, Defending the Choice 
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Suppliers 

e Those customers don’t live with the results 
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¢ CMMI - Developed by Systems / Software 
/ Defense Professionals 
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¢ CMMI - Systems Engineering, Software 
Engineering, Integrated Product and Process 
DY-W/<1(e) ola alc) alemretale mele) @)e)/(o)mrelelelaeale 

¢ ISO 9001 - Generic 

¢ CMMI - “Improving Processes for Better 
Products” 


¢ ISO 9001 - “Quality management systems — 
Requirements” 
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Requirements 
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e ISO 9001 - Requirements only. Same for 
all companies, industries, disciplines 
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Assessment Results 


¢ CMMI — Determines Maturity 
e ISO 9001 — Based on Conformity 


e Size Does Matter 
— CMMI - 729 Pages 
—ISO 9001 - 33 Pages 
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Net Effects 
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CMMI easier to claim (considering 
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CMMI more appropriate for software, 
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e Customer’s requirements 
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¢ Competitor’s status 

¢ Business objectives 
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New Processes and Records 
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“Claiming” Process (through assessments) 
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Approaches (and Defenses) 
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ISO 9001 Registered, Addressing Specific 
Guidance Documents 


ISO 9001 Registered, Developed for 
Selected CMMI Maturity 


CMMI SCAMPI Assessed 
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From the very first glance at the CMMI standard, it is obvious it is more 
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The Unseen Benefits of t 
Change Request Proces 


Scott Banks 
Mark Pellegrini 


Electronic Systems Laboratory 
Georgia Tech Research Institute 
Georgia Institute of Technology 


Georgia || Research 
Tech| Institute 
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of Technology 


Georgia Tech Research Institute (GTRI) Overview 


Georgia || Research 
Tech 


Unit of the Georgia Institute of Technology 

1200+ employees 

70% of research employees hold advanced degrees 
Wide variety of products 


Customers include federal and state government; 
and industry 


Competitively bid projects range greatly in size and 
duration 


More Info: http://www.gtri.gatech.edul/ 


Institute 


Georgialnsititute 
of Technology 


Topics to Cover 


¢ What is a problem report 

¢ What a problem report is not 

¢ Why problem reports are a good thing 

¢ How problem reports fit into your development process 
¢ Problem report lifecycle 

¢ Types of information to collect 

¢ Problem report metrics 

¢ Summary 
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What is a Problem Report? 


¢ PR, SPCR, Bug, OCR, CR, EPR, ECR... 
¢ Project requirement 
¢ Creates a paper trail 


¢ Documents changes at any point in development 
cycle 


¢ Easy PM tool (How are we doing?) 
¢ Part of release documentation 


¢ Business development tool 


Gecrula Research 
Vy Ornette 


What is a Problem Report? 


¢ Part of the CMMI v1.2 Configuration Management Support Process 
Area ML2 


¢ Specific Practice: SP 2.1 Track Change Requests 
¢ Typical Work Product: Change Requests 
¢ Subpractices: 
1. Initiate and record change requests in the change request database. 


2. Analyze the impact of changes and fixes proposed in the change 
requests. 


3. Review change requests that will be addressed in the next baseline 
with the relevant stakeholders and get their agreement. 


4. Track the status of change requests to closure. 


Georgia || Research 
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What a Problem Report is Not 


¢ Not optional 
¢ Not a personal performance indicator 
¢ Not a singular project tool 
¢ Not a contest 
¢ Not only for S/W 
¢ Documentation 
¢ Systems/Hardware 
¢ COTS 
¢ Subcontractors 


e Process 
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Why Problem Reports are a Good Thing 


« Great communications tool 


Provide a status snapshot 

¢ Manage clients and subcontractors 
¢ Issues don’t get lost 

¢ Help control the product 

¢ Bring new hires up to speed 

¢ Business development 


¢ Help make informed decisions 
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How Problem Reports Fit Into the 
Development Cycle 


¢ At what point during the process should 
problem reports be written 


¢ Who should see problem reports 
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Problem Report Lifecycle 
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Types of Information to Collect 


Unique ID 


¢ Description 


State of problem report 


¢ Functionality or application 


Where in the process it was written 


¢ Severity 


Priority 


¢« Dates 


Estimations 
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Problem Report Metrics 


¢ Open vs. closed 

¢ Where defects are introduced 

¢ Closure rate 

¢ Days to verify 

¢ Open per function or application 


¢ When defects are found 
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Problem Report Closure Rate 


|—-+— Total Number of Problem Reports —s— Open Problem Reports —.. Closed Problem Reports | 


Number of Problem Reports 
B 
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Feb Mar 
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Problem Report Graphs - Where Found 
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Problem Report Graphs - Where Found 
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Problem Report Graphs —- Status 


5% G Submitted 
@ Hold 
O Reject 


12% 10% 


20% 


O Research 
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mw Closed 
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Problem Report Graphs - Status 


8% 


f Submitted 
w Hold 

oe O Reject 

O Research 
m Fix 

B Verify 


5% Tay: 70/ 3% m Closed 
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Problem Report Graphs — Severity 


5-Enhancement 


4-Inconvenience 


3-Moderate 


2-Severe 


1-Critical 
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Problem Report Graphs -— Priority 
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Excuses 


¢ Takes more time to document the change than to 
actually make it 


¢ This will reflect poorly on my work 


¢ | don’t want the customer to know what our 
problems are 


¢ Stop writing problem reports so we can release 
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V of Technology 
Summary 


¢ Simple way to gain project insight and 
make informed decisions throughout 


¢ Good project communications tool 

¢ Helps keep things under control 

¢ Don’t misuse 

¢ “Tool” should support your process and 
needs. Consider things like web 


interface, queries, report generation, 
customization or integration 
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Questions? 
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Contact Information 


« Scott Banks 
scott.banks@gtri.gatech.edu 


¢ Mark Pellegrini 
mark.pellegrini@gtri.gatech.edu 


¢ More Info about GTRI: 
http://www.gtri.gatech.edul 
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Measurement & Analysis CMMI™ 
Process Areas To Achieve More 
Successful Software Projects 


Daniel D. Galorath 


galorath@galorath.com 
310 414-3222 x614 
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Integrating Planning, Control, Measurement & Analysis CMMI™ Process Areas 
To Achieve More Successful Software Projects 


Project Planning and Project Monitoring and Control, two CMMI® Level 2 
Process Areas, are essential to Software Project Management. Prior to project 
Initiation, the relationship between these two process areas tends to be strong. 
Unfortunately, the status quo tends to exhibit a breakdown in this relationship 
once the project has begun. The result is an open-loop situation: we know where 
we planned to be and where we are; however, we don’t know where and when we 
will likely end up. 


= This paper first reviews the fundamentals of Parametric Estimating and 
Performance Measurement 


= and then proposes an integrated process uniting Project Planning, Project 
Monitoring and Control, and Measurement and Analysis. The benefits of this 
integration are an increased probability that: 1) a project’s plan is achievable 
(i.e., it honors the laws of software project dynamics and keeps from “death 
marches”); and 2) actual performance matches reasonably-expected (planned) 
performance; (i.e., that a project is delivered on time and within budget); and 3) if 
there are troubles that there is an early warning of trouble and possibilities for 
managers to remediate. Finally, the paper chronicles, through the use of selected 
artifacts, an actual project employing this integrated approach. 
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SEER CMMI Key Process Areas Require 
veewonwe™ Estimation With Feedback 


Estimate 
& 


oy 
eg 
eo 
¥ s Project Monitoring and Control 
& x ¢ Monitor Project Against Plan 
we * Manage Corrective Action to Closure 


Monitor 
& 
Control 


Analyze 


Measurement and Analysis 
¢ Align Measurement and Analysis Activities 
e Provide Measurement Results 


line(s) ® 
cual Actuals 
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Fundamentals of Software 
Estimation 
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SEER Poor Estimates Effects on 
eee Projects 


= Inaccurate estimates can have significant impact on project 
success: 
Poor implementations 
Critical processes don't scale 
Emergency staffing 
Cost overruns caused by underestimating project needs 
= Lack of well defined objectives, requirements, & specifications, 
results in creeping scope resulting in: 
Forever changing project goals 
Frustration 
Customer dissatisfaction 
Cost overruns and missed schedules 
Project Failures 
= Incorrect estimates / bad plans are a root cause of subsequent 
program risk 
Estimating & Planning are key to software project success 
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‘ We Can’t Afford Not To Spend Time and Money On 
SEER 


Viable Estimates and Project Plans 
Adapted From Aspire S. Hunt 


GAtLORATH 


Apollo 

Most Apollo program faults were software related 
Loss of NASA Spacecraft 

FORTRAN statement error caused loss of 1st NASA probe to Venus 
After success with Ariane 4, the maiden flight of Ariane 5 ended 
in destruction when design defects in the control software were 
revealed by the faster horizontal drifting speed of the new 
rocket. 
= Financial & Program Impacts 


Airbus A310 45,000 hours expended against software development 
(20,000 hours budgeted!) 


American Airlines lost nearly $1billion due to software faults 


In 1991 after 3 lines of code (in millions of lines of code in a signaling 
program) were changed, telephone systems in California and the Eastern 
Seaboard went off line 


= Loss of Life 


Gulf War 1 Patriot Missile tracking failure due to 0.36 second software 
driven clock. 


And so on....... 
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SEER An Estimate Defined 


= An estimate is the most knowledgeable statement 
you can make at a particular point in time 
regarding: 
-| Effort / Cost 
“| Schedule 

-| Staffing 

‘| Risk 

(1 Reliability 
= Estimates more precise with progress 


mA WELL FORMEDESTIMATE ISA 
DISTRIBUTION 


Density 
Confidence 
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Effective Technology ___, 
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Effort 
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& Defects 
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M cot mes” 


Constraints 


Software 
Development |===> | Delivered 


Process 


Software 


7 
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Start A 
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Effort 
Progress 
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SEER SEER-SEM Provides Staffing Plans that Avoid “Death 
catoratn Marches” and Failed Projects Applying “Brooks Law” 


Min Time 
Staffing 


Unaccomplishea 
Work Level 


Staffing 


Staff Level (FTE people) 


Actual 


1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 
Elapsed Calendar Time (months) 


@ Effective Staffing |= Staffing Beyond Plan m@ Overstaffed Mm Understaffed 
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* SERR 10 Step Software Estimation Process 
4 GALORATH Supports Software Best Practices 


1. Establish estimate scope 


} (2. Exabls technical 
baseline, groundrules 


& assumptions 


3. Collect data 
-<e 


5. Prepare baseline 
estimates 


6. Quantify risks & 
risk analysis 


i 


7. Review, verify, 
validate estimate 


8. Generate a 
project plan 
9. Document estimate 
& lessons learned 
10. Track project through- 
out development 
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SEER Step One: Establish Estimate Scope 
ee and Purpose 


= Define and document estimate expectations, scope & 
Purpose 


-| Provides a baseline against which to gauge the effect of future 
changes 


Reduces misunderstandings & contradictory assumptions 
ms Estimate should be considered a living document 


As projects change, data changes or new information becomes 
available, it should be documented and factored into the estimate 
in order to maintain the project’s integrity 
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SEER Define What’s Included in the Estimate 


GAtLOR ATH 


= Development effort breaks down into... 
a What functionality will be developed 


= How will it be done? 
Activities & phases 


= Who will do the work? 
Personnel labor categories 
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SEER Step Two: Establish Technical 
cstorsts Baseline, Groundrules, & Assumptions 


= Functionality included in the estimate or range must 
be established 


lf detailed functionality is not Known, groundrules and 
assumptions state what is and isn’t included in the estimate. 


Issues of COTS, reuse, and other assumptions should be 
documented as well. 

= Groundrules and assumptions form the foundation of 
the estimate 


although early at early stages they are preliminary and therefore 
rife with uncertainty, they must be credible and documented 


Review and redefine these assumptions regularly as the estimate 
moves forward. 


© Galorath Incorporated 2006 


SEER Technical Baseline 


ALORA TH 


= Describe each work element 

-| Core functionality 

-| Key qualitative modeling inputs 
Size, if available 


Identify multiple builds/releases (if any) 
Reference or excerpt available documentation 
Provide rationale & justification for BOE 
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SEER Ground Rules & Assumptions 


GAtLORATH 


= Groundrule: given requirement of the estimate (e.g. 
software must support windows and Linux) 
Be sure to provide source 
= Assumption: assumed to scope estimate 
Be sure to provide source and substantiation 
a What’s known, what’s unknown 
a Anything relating to scope 
What’s included, what’s excluded 
a Anything relating to modeling inputs 
Who you interviewed and when 
What you learned 
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SEER Step Three: Collect Data 


GAtLORATH 


= Software Data Collection Process key considerations 
Motivate potential data providers to participate 


Avoid nondisclosure agreements containing clauses requiring 
exclusivity or destruction of data if you can 


Provide data collection forms and instructions beforehand, in both 
hard copy and electronic formats 


Provide clear definitions but recognize providers may not read 
them 


Identify which data are required, highly desirable or desirable 
During the face-to-face interview confirm data is realistic and valid 
Grade to indicate confidence 


Normalize data via well-documented process & Keep both the 
raw and normalized data 
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SEER Data Collection 


GAtLORATH 


= Begin with available documentation 
SDP, TRD, SRS, COO, ORD, CARD, BAFO, BOE, SOW 
| A-level Spec, Program Schedule, Contractor Presentations 


= Perform interviews where & when possible 
Development contractor(s) 

Program office personnel, FFRDC engineers 
Preparation is paramount! 

Use a questionnaire, sent in advance 

Phraseology determines answers 

Confirm key definitions 

Be willing to challenge, point out inconsistencies 
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SEER Step Four: Software Sizing 


GAtLORATH 


= Spend Time on sizing 
-| Include rework that will be required to develop the product, 


= Estimate /east likely most range. 


= Some widely used methods of estimating product 
size are: 

Expert opinion 

Analogy 

Formalized methodology 


Statistical sizing — Provides a range of potential sizes that is 
characterized by /east, likely, and most 
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SEER _ Steps to Estimating Software Size 
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= Use a software sizing method that is consistent and repeatable 
Reestimate as specs change 
Establish baseline definition of the size metric & identify normalization 
process if different from the definition chosen 
1. Define sizing objectives 
@ Are you trying to describe the size of individual computer programs, plan 
major milestones in the estimation process, or adjust for project 
replanning? Varying granularities of sizing detail will be appropriate 
2. Plan data and resource requirements of the sizing activity 
3. Identify and evaluate software requirements, a set of software 
specifications that are as unambiguous as possible 
4. Use several independent techniques and sources and put 
findings in a table 
5. Track the accuracy of the estimate versus actuals as the project 
progresses, and reestimate the product size periodically with 
actual data. 


© Galorath Incorporated 2006 


SEER Galorath Size Estimation Methodology 
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Apply Size Estimation Methodology 


Evaluate all sources of software size... 


Pre-cx corsiaors | 


SLIME 
size (rework ) 


Total Size Estimates Least |Likely |Most | 


Multinte size  ExPettYudgement__|—12000| 15500] 17000} 
Multiple Size [Relevant Range by Analog 32540 J) Viable size 


estimates range 


SizingDatabase | 8000] | 46000} 
Functional Analysis |_| 27540] 


SEER-AccuScope |__ 15450] 22650] —-29850 | 
Delphi Analysis 15788 19750 22713 


Expert SEER-Accuscope Sizin 
pudvementt analysis databases 


Functional Counts for \nalovres 


J 
anaivsis j pre-caisling i — —_ 
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SEER _ Step Five: Prepare Baseline Estimate 


1. Trained, experienced, and skilled people should be 
assigned to size the software and prepare the 
estimates 


2. It is critically important that they be given the proper 
technology and tools 


3. Project manager must define and implementa 
mature, documented, and repeatable estimation 
process 
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SEER Estimation Approaches 


GAtLOR ATH 


Guessing (which is not recommended) 
Using existing productivity data exclusively (also not recommended) 


Bottom-Up Estimating 
Bottom-up (a.k.a. grassroots) estimating, decomposes the software to its lowest levels by 
function or task and then sums the results 
Can be effective for smaller systems and where estimates are carefully produced 


Parametric Software cost models 
Provide means for applying a consistent method for subjecting uncertain situations to rigorous 
mathematical and statistical analysis 
When used properly they are more comprehensive than other estimating techniques and help 
to reduce the amount of bias that goes into estimating software projects 
= Delphi Based Estimates 
Several expert teams or individuals, each with an equal voice and an understanding up front 
that there are no correct answers, start with the same description of the task at hand and 
generate estimates anonymously, repeating the process until consensus is reached 
= Activity-Based Estimates 
Begin with requirements & size then define the required tasks, which will serve to identify the 
overall effort that will be required 
Typically Major cost drivers focused on non-coding tasks included in effort estimate 
Build a task hierarchy (WBS) that represents how the work will be organized and performed 
This work breakdown structure can be the backbone of the project plan 
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SEER Generate the Estimate 


byG ALOR ATH 


Using your chosen methodology and tool, do a first 
run 


Never report preliminary results! 


Focus on the inputs 
‘1 Verify completeness 
“1 Verify accuracy 


Focus on the outputs —acaieicanas: 


‘1 Sanity check for 
Rank 
reasonableness, 6 fpiicallb Laas Sei 
completeness Pt ee 
3. Special Display Requirernents 
a What’s driving the estimate? 4 cevicomenr system voistiny 
1 SEER-SEM top ten paramete : ee Gt 
m Use “fresh eyes” to review 7. Process Volatility 
8. Dev Sys Exper/Compl 
(1 Ask a colleague for help Gate he 
OC Set aside overnight 10. Rehost from Development to Target 


Vea gr ne VE Sarn 
oe re 


Data Analyzer 
== 


ween ee ve 
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SEER  —_—Update/Refine the Estimate 


Optional or more specific information may become 
available in the future 


Follow configuration control guidelines to keep 
estimates straight 

Change log 

Naming conventions 

File storage 


= Maintain estimating integrity - eschew artificially 
driving the estimate to a predetermined result 
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SEER Step Six: Quantify Risks and Risk 
——— Analysis 


Risk can produce loss of time, or quality, money, control, 
understanding, and so on. 

Loss associated with a risk is called the risk impact 
Approximate the probability that the event will occur 


Risk probability: Likelihood the risk, measured from O (impossible) to 1 (certainty) 
When the risk probability is 1, the risk is a problem since it is certain to happen. 


Determine how risk can be mitigated 

Risk control involves a set of actions taken to reduce or eliminate a risk. 

=» Risk management identifies & addresses internal & external potential 
threats 


Problems associated with sizing and estimating software potentially can have 
dramatic negative effects. 


If problems can be foreseen & causes acted upon in time, effects can be mitigated 
= Although cost, schedule, and product performance risks are 
interrelated, they can also be analyzed independently 
Risks must be identified as specific instances in order to be manageable 


Statistical risk/uncertainty analysis should be a part of schedule & effort estimation 
process 
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SEER Understanding Risk and Uncertainty is 
ysaconats Essential To Project Management 


S§ Risk and Goals - Program: Analyst Support System 


Parameters Goals Confidence 


Analyst Support System 


Schedule Goal (months) 

Total Development Cost Goal 1,000,000 
Effort Goal (hours) 0 
Max Staff Goal 

Effective Size Goal 

Defects Goal 


Effort Risk Schedule Risk 


Hours (in K) Analyst Support System Months Analyst Support System 
30 = CQzture nS <Ctr-shift-P> | ie 
48 


24 
18 36 


12 24 


= 


0 —— 0 ———_ 
» (2 1% 10% 20% 30% 40% 50% 60% 70% 80% 30% 99% » || 1% 10% 20% 30% 40% 50% 60% 70% 80% 390% 99% 
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SEER Step Seven: Estimate Validation and 
— Review 


Ideally, validation performed by one who was not 
involved in generating the estimate 


Assess estimate assumptions 
= Ensure groundrules are consistent applied 


= Rigorous validation process exposes faulty 
assumptions, unreliable data and estimator bias 

Provides clearer understanding of the risks inherent in your 
projections 
Having isolated problems at their source, you can take steps to 
contain the risks associated with them, and you will have a more 
realistic picture of what your project will actually require to 
succeed 

= Failing to validate the estimate may result in much 

greater downstream costs, or even a failed project 
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SEER Compare Parametrics With Metrics and 
—— Sanity Checks 


Works with common  eceeteaiah elie 


File Edit View Chart Options Window Help 


repository osaeeer 
Shows actual data, ranges, Effective Size vs Estimated Effort 


Data Points 


ue e Historical Data 
and correlations F oe a Current Estimate 
7 a Reference Estimate 
Plots SEER-SEM estimates 30, + Basoine Estimate 
a Forecast Estimate 
“oe Trend Lines 
.-.-- History Trend (mean) 


rP=0.94 
y = 252.1281x"0.8062 
— -History +/-16 
— -History +/-26 
— -History +/-30 
Benchmark +/-16 
Filter 
Platform = Avionics, 
Manned Space 
Observations = 85 


and contrasts with data 


Scatterplot Plus Chart Properties 


Inputs and Controls | Estimate Data | Format Axes | Show/Hide Points | 
DataSource | History Display Options | Benchmark Display Options | 
X&Y Metrics Filter 


Effective Size (ESLOC) (Log) 


c Downselect based on current estimate's knowledge 
base settings 


Jt 


100 1000 
Estimated Effort (Labor Months) (Log) 


© Display full range of project types 


@ Manually select project types to be included 


Fields Selection 
SG 
Clear All Filters 


Financial Processing a !No Knowledge 
Ground-Based Mission Critic Artificial Intelligence 
Ground-System Non-Critical Business Analysis Tool 
Internet Development Command/Control 


Camm iniestiane 


No items selected equals ALL items selected 


Save Configuration Apply To Chart Close Cancel Help 


“In God we trust, 
all others bring data.” 
~W. Edwards Demin 
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SEER Step Eight: Generate A Project 
eee Plan 


includes allocating estimate cost & schedule & allocating the 
cost and schedule to a function and task-oriented work 
breakdown structure 
Issues 
Inexperienced managers may not properly evaluate decisions effects 
over the long run. 
a lack necessary information 
= believe time to develop the information will make the project suffer 
Other managers make decisions based on what they think higher 
management wants to hear 
Good manager understands project realities even if it is not what higher 
management wants. 
a Job is to explain the reality in language his managers can understand 
=» Problem managers although they may mean well, either lead a project to an 
unintended conclusion or, worse, drift down the road to disaster. 
= Software management and planning problems have been 
recognized for decades as the leading causes of software 
project failures 
bad management decisions 
Incorrect focus 
destructive politics 
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SEER Revitalizing the Software Aspects of 
= Systems Engineering 


Memorandum of 13 June 2004 from Undersecretary of 
the Air Force: “Revitalizing the Software Aspects of 
Systems Engineering” 
= Number 1 (of 10) recommendation: High Confidence 
Estimates 
“Estimate the software development and integration effort (staff 
hours), cost and schedule at high (80-90%) confidence.” 
= e¢Number 2 recommendation: Realistic Program 
Baselines 


“Ensure cost, schedule and performance baselines are realistic 
and compatible. ... The program budget must support the high 
confidence estimates for effort (staff hours), cost, and schedule.” 


= Number 3 recommendation: Risk Management 


“Continuously identify and manage risks specific to computer 

systems and software as an integral part of the program risk 

management process. Ensure the risks, impact, and mitigation 

plans are appropriately addressed during program and portfolio 
© Galorath Incorporated 27Q\/| eWS.” 


SEER Expand Product WBS to Task level 
—_— Plan 


Transforms Microsoft Project into a powerful tool 
for planning software development projects. 


= Automatically constructs a 
complete project plan 
| with relatively few inputs 
| or directly from your SEER- 
SEM project estimate. 
m You can create custom life cycle 
templates. 


= You can customize labor Es = mh SEER-SEM Client 
categories to reflect the way that 
your organization assigns tasks 


to departments or labor sea en 
. [fiers ¥ 

categories to accurately plan ee 

staff allocation for a project. Se 


Go ba ayo) a 


a 


8 
MS Project w/ SEER-SEM Client : : 
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SEER Step Nine: Document Estimate and 
—e Lessons Learned 


Document upon estimate complete AND project complete 
document the pertinent information 
record the lessons you learned. By doing so, you will have evidence that 
your process was valid and that you generated the estimate in good faith, 
and you will have actual results with which to substantiate or calibrate 
your estimation models. 
Be sure to document any missing or incomplete information and 
the risks, issues, and problems that the process addressed and 
any complications that arose 
= Document key decisions made during the estimate & results 
= Document dynamics that occurred during the process e.g. 
interactions of your estimation team 
interfaces with clients 
trade-offs made to address issues identified during the process. 
= Conduct a lessons-learned session as soon as possible after the 
completion of a project while the participants’ memories are still 
fresh. 


= Every software project should be used as an opportunity to 
improve the estimating process 
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SEER Step Ten: Track Project Throughout 
=_— Development 


Refining Estimates throughout Project 

Once a project has started, use estimates as a basis 
for performance measurement & project control 
Monitor actual effort & duration of tasks and/or 
phases against planned values to ensure you have 
the project under control 

= Applying earned value techniques along with 
parametric estimation can help ensure successful 
projects 


= Evaluate defects & growth in addition to simple 
earned value 
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SEER Earned Value as a Root Level Software 
———- Management Technique 


Many think of earned value as a high level program 
control metric 
But Earned value analysis is most valuable when the 
units of work are small (1-2 week activities) 
Use binary earned value... 

Either not done (earned O value) or 

Complete (earned full value) for the task 
= Galorath has used earned value at the root level as a 
core technique for saving failing software projects 
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SEER 


GA £-O: RR A Tin 


The main concern of EVM is what has been 
accomplished in a given time and budget, versus 
what was planned for the same time and budget 


A project is generally deemed healthy if what has been 
accomplished is what was planned, or more 

A project is deemed unhealthy if accomplishment lags 
expectations 


a Definition: Earned value = budgeted value for the 


work accomplished (what you got for what it cost 
you) Healthy Unhealthy 
udget $ udget 


EV 


Time = Now Time = Now 
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GA LOR ATH 


CMMI Process Areas For 
at lalaliace Mm Celali cela lare m7 
Control 


© Galorath Incorporated 2006 
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SEER CMMI Key Process Areas Require 
veewonwe™ Estimation With Feedback 


Estimate 
& 


oy 
eg 
eo 
¥ s Project Monitoring and Control 
& x ¢ Monitor Project Against Plan 
we * Manage Corrective Action to Closure 


Monitor 
& 
Control 


Analyze 


Measurement and Analysis 
¢ Align Measurement and Analysis Activities 
e Provide Measurement Results 


line(s) ® 
cual Actuals 
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|, SEER _—‘ The CMMI Process Areas 


tyG ALORATH Source: CMMI Tutorial 


Category Process Area 


1. Project Planning 
2. Project Monitoring and Control 
Supplier Agreement Management 
Integrated Project Management(IPPD) 
Integrated Supplier Management (SS) 
Integrated Teaming (IPPD) 

3. Risk Management 

4. Quantitative Project Management 


Project 
Management 


Support Configuration Management 
Process and Product Quality Assurance 
5. Measurement and Analysis 
Causal Analysis and Resolution 
Decision Analysis and Resolution 
Organizational Environment for Integration (IPPD) 


Engineering Requirements Management 
Requirements Development 
Technical Solution 
Product Integration 
Verification 
Validation 


Process Organizational Process Focus 
Management Organizational Process Definition 
Organizational Training 
Organizational Process Performance 
Organizational Innovation and Deployment 


© Galorath Incorporated 2006 


SEER 


YGAtLtLORATH 


ER-PPMC: Performance Mode 
ignificant Overrun A parrent 


ABT fate Why 
108.57 


Data Value 


Development Schedule Months 
Development Effort Months 4 


Baseline Estima 


16,502 
1,595,906 

2/02/04 

7/31/05 


Development Effort Hours 

Development Base Year Cost be 
Start Date 

Completion Date 
Effective Lines Only 
Effective Functions Only 
Effective Size 


Pb 


PERFORMANCE INDICES 

Cost Performance Index(CPh) 
schedule Performance Index (SPI) 
Composite Performance Index (XPI) 
Time Performance Index (TP 


VARIANCE 
Cost Variance (CV) (Hours) -1,038 
schedule Variance (SV) (Hours) 
Time Variance (TV) (Months) 


ACCOMPLISHMENTS 
Hours thru Latest Snapshot 5,200 
Cost thru Latest Snapshot 502,895 
Earned Value thru Latest Snapshot 
Earned Value % thru Latest Snapshot 
Planned Hours thru Latest Snapshot 


Lo? WALA ul mg. AA 


Projected Productivity Lines/PM Terminology Standaig e 
Sohal is yi See Forecasting Method 


[~ Show Baseline WBS Elements 
Show Current Estimate in PPMC Charts 


PPMC Report Options OK Cancel 
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SEER-PPMC Parametrics Combined With Traditional EVM 
Captures Software Project Trouble More Completely 


SEER-PPMC: Basic (EVM) Mode 
Overrun Not as Apparent 


Data Value 


Development Schedule Months 
Development Effort Months 
Development Effort Hours 
Development Base Year Cost 
Start Date 

Completion Date 

Effective Lines Only 

Effective Functions Only 
Effective Size 


PERFORMANCE INDICES 

Cost Performance Index (CPI) 
Schedule Performance Index (SPI) 
Composite Performance Index (XPI) 
Time Performance Index ({TPh 


VARIANCE 

Cost Variance (CV) (Hours) 
Schedule Variance (SV) (Hours) 
Time Variance (TV) (Months) 


ACCOMPLISHMENTS 

Hours thru Latest Snapshot 

Cost thru Latest Snapshot 

Earned Value thru Latest Snapshot 
Earned Value % thru Latest Snapshot 
Planned Hours thru Latest Snapshot 
Projected Productivity Lines/PM 
Projected Productivity Functions/PM 


Baseline Estimat 


16.77 
93.73 

14,247 
a 1,377,824 
6/ 2/02/04 


4% 
8% 
8% 
8% 


5,200 
502,895 
4,162 
31.51% 


Terminology 


Forecasting Method 7 


[ Show Baseline WBS Elements 
! Show Current Estimate in PPMC Charts 


PPMC Report Options | OK Coa | 


SEER CMMI Project Planning Requires Schedule & Effort 
catorats# Estimation & Planning — Context Source CMMI Tutorial 


Planning — 


Develop a Project Plan 


Establish 


: Plan Plan for 
the Budget Identify for Data Project 
and Project Risks Management Becencce 


Schedule 


Plan Establish Plan for 
Stakeholder the Project Needed 
Involvement Plan Knowledge 


and Skills 


Project Plans 
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SEER CMMI Estimation Requires Effort, 
»sacoratsx SChedule, Scope — Context source cmmi Tutorial 


Establish Estimates 


Establish 
Estimates of 
Work Product 
and Task 


Planning 
ean = 


Estimate 
the Scope 
of the Project 


Determine 
Estimates 
of Effort 
and Cost 


Define 
Project 
Life Cycle 
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SEER CMMI Project Monitoring & Control — Context 


GALORATH Source CMMI Tutorial 
(SS SS SS eS SS SS eS eS ee ee ee “ = M ————— | 
Monitor Project Against Plans Saiecaue nclisne 
to Closure I 


biebblaad Monitor Monitor Conduct 


Project ; ° 
Bee ee oe) none Analyze 
Parameters SKS Issues 


Monitor 
Data 
Management 


Conduct 


Monitor 
Commitments 


Corrective 


progress 
Reviews 


Actions 


Manage 
Corrective 
Actions 


Project 
Planning (P 


I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I Take 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 


>) Project Plans —— 
“4 nn 
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SEER The First Step To EVM Is A Workable 
an! Plan for Cost & Schedule 


“...Using EVM properly puts a 
considerable burden on team 
members to plan and schedule 
projects -- skills often in short 
supply at federal agencies” 


»»» Norman Enger, OMB 
http://www.fcw.com/iarticle91403-11-14-05-Print 
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SEER CMMI Measurement & Analysis — 
soon Context: source CMMI Tutorial 


Align Measurement Analysis Activities 
Specify 


I 
I 
I 
Establish Specify rier Specify 
Measurement hieaciice = Sorc Analysis ; 
jecti Pr r I 
Objectives Brecedirce ocedures | 
I 
——— ) 
M remen ———————e 
Measurement Objectives easurement Procedures, 
Repository 
—__ $$ _—_ Tools 
Measurement Measurement Indicators 
‘eal ~ fe 
'Provide| Measurement Results s 


Collect 
Measurement 
Data 


Store 
Data & 
Results 


Analyze 
Measurement 
Data 


Results 


I 

I 

I 

I . 

1 {Communicate 
I 

I 

I 

I 
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SEER Risk Management Source: CMMI 
— Tutorial 


= Purpose: 


| Identify potential problems before they occur, 
so that risk handling activities may be planned 
and invoked as needed across the life of the product or project to 
mitigate adverse impacts on achieving objectives 


= SEER Provides the cost & schedule impacts of such 
risk 

= According to Mike Evans “We can tell them they have 
a problem. SEER can quantify what the impact is” 
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SEER CMMI Key Process Areas Require 
veewonwe™ Estimation With Feedback 


Estimate 
& 


oy 
eg 
eo 
¥ s Project Monitoring and Control 
& x ¢ Monitor Project Against Plan 
we * Manage Corrective Action to Closure 


Monitor 
& 
Control 


Analyze 


Measurement and Analysis 
¢ Align Measurement and Analysis Activities 
e Provide Measurement Results 


line(s) ® 
cual Actuals 
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SEER Performance Measurement: Earned Value 
catorath Analysis Can Make Projects More Successful 


= “Earned Value Analysis” is an industry standard 
way to: 

measure a project’s progress 

forecast its completion date and final cost 

provide schedule and budget variances along the way. 


= Provides consistent, numerical indicators with which 
you can evaluate and compare projects 


= Earned value (Performance Measurement) techniques 
are just as viable using effort for root level project 
management 
Effort Value — units of labor; e.g.: 
=m person-hours, staff-hours, effort-hours, labor-hours 
= person-months, staff-months, effort-months, labor-months 
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SEER Earned Value Measures Support CMMI 


———ans Tracking & Control 
Source: PMBOK & Galorath 


Baseline Budget — Budget at 
Completion (BAC) 


Planned Value — Budgeted 
Cost of Work Scheduled Buriget 
(BCWS aka PV) Completion 
7 Planned Value: Budgeted / ni 
cost for work scheduled to be 
completed up to a given 


Actual 
Costs. 


ue 
§ 
= 
o 
= 
5s 
a 
K 
Lt 


point in time ro 
= Earned Value — Budgeted we ee 
Cost of Work Performed al ——|_ 
(BCWP aka EV) 


| Progress: Budgeted value for 
work actually completed 
during a given time period. 
= Spent — Actual Cost of the 
Work Performed (ACWP aka 
AC) 
| Actual cost (AC) - Total cost 
accomplishing work during a 
given time period. 


Figure 7-7. Mlustrative Graphic Performance Report 
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SEER Looking at Progress Over Time 


byG ALORATH 


See how actual and earned 


See CPI, SPI and TP! trends 


effort measure up to the over time 


baseline plan 


% of Baseline Plan 


te Data Analyzer 


As of 9/16/05 - Design Review 


Data Analyzer (Cumulative) 
As of 9/16/05 - Design Review 


Current Est ——= 
Eamned Value — 
Actual Spent 
Baseline Plan 

Forecast End Date: 8/30/07 $@ 

Forecasting Method: Cum Performance 

Time Now: 9/16/05 WY 


0.00, 


cormonee 2-05 3-05 4-05 5-05 6-05 7-05 8-05 9-05 10-05 


2-05 6-05 10-05 2-06 6-06 10-06 
Baseline EAC: 56519 Hours; 30.98 Months 


CPI = Cost SPI = Schedule TPI = Time 
Performance Performance Performance 


Click on any Index; Index; Index; 


month to see 
specific The ratio of The percent of The ratio of 
information earned to actual the investment elapsed time 
cost of work that has been from start to 

performed completed plan & start to 


time-now 
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SEER 


byG ALOR ATH 


Heath and Status Indicator 
shows status and trends from 
the previous snapshot 


Thresholds are user definable 


Sf Health & Status Indicator 


Schedule Time 
Variance Variance 


Analyst Support Sy... 
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Defects and Growth Impact Software 
Process 


Track defect 
discovery and 
removal rates 

against expected 
rates 


Sf Defects Tracking 


Data Analyzer 


Defects 
2000 


Baseline Defects Inserted 
Baseline Defects Removed 
Actual Defects Reported 
Actual Defects Removed 


Increased defect 
reporting rate 
shows a 
worsening trend 


Cost Size 


Variance Growth Defects 


SEER 


rere Te Parametric Project Monitoring & Control 
Provides Performance Measurement aspects of ANSI/EIA-STD-748 


Adds Performance Measurement Bd Fronted Lie 


(Earned Value) methods to SEER- . 
SEM sind CPU 4 (Cumulative) 


Accepts progress & expenditure 
inputs 

Provides cost, schedule, and time 
variances 


Provides cost, schedule, & time 


indices “105 305 505 705 905 1105 106 306 506 


= Performance-based cost & 
schedule Estimate at competion 


ForecastEnd Date: 6/18/07 ¢ 
Forecasting Method: Performance 
Last Update: 5/22/06 FF 


15 HS HS 765 HS 1105 106 306 506 706 906 1106 107 SOT 507 707 307 
Baseline At Complete: 51173 Hours; 31.56 Months Months 
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SEER 


byG ALOR ATH 


Heath and Status Indicator 
shows status and trends from 
the previous snapshot 


_ eIncluding Size Growth and Defect 
Discovery/Removal Rate 


-User defined control limits to 
control the transition between 
red-yellow-green 


4+ Health & Status Indicator 


Schedule 
Variance 


Data Analyzer 
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Time 


Variance 


Defects & Size Growth Are Additional 
Core Metrics 


Track defect 
discovery 
and removal 
rates against 
expected 
rates 


4: Defects Tracking 


Data Analyzer | 


As of 7/09/06 - Code Review 


Estimated Defects Reported 
Estimated Defects Removed 
Actual Defects Reported 
Actual Defects Removed 


Oa i a 


Increased defect 
reporting rate 
shows a 


Defects worsening trend 


Cost 
Variance 


SEER “Dollarizing” Risks 
ies Source J. Hamaker 


Risk Areas 


Requirements 


= Technical / Performance / Estnating Variance 
Engineering 
=» Schedule Nagas 


Cost ee ~ 
Safety | 


hecho TOTAL PROJECT = Estimates are really 
p probability distributions 


Sources of uncertainty 
‘| Project uncertainties 


| / 
$ $ _ 

| ; . eee ‘| Model variance 
$ 


Cumul 
a 


Lows via modeling optimistic assumptions 


Highs via modeling pessimistic assumptions ‘| Cost/ risk analysis quantifies 
budget / plan with acceptable 
confidence levels 
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SEER ‘Parametric Modeling Supports 
| CMMI 


= Parametric Cost Estimating Methods 


Estimating methods based on physical or performance 
characteristics and schedules of the end items 


= Parametric Estimation Model Definition 


Mathematical representation of parametric estimating 
relationships that provides logical, predictable correlation between 
the physical or functional characteristics of a system, and the 
resultant cost, schedule, and other attributes of the system. 
=» Comprised of cost estimating relationships (CERs) and other 
parametric estimating functions, e.g., cost quantity relationships, 
inflation factors, staff skills, schedules, etc. 


= Provides logical repeatable relationship between input variables and 
resultant cost 
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'_ SEER Example Simple Cost Estimating Relationship 
eiainialiiidi (CER) Shows Effort Based On Size 


4 Scatterplot Plus 
Eyre Ewe Yew Chace Octins Window. pee 


(O68 ®@4? SLSHB 


Development Effort Months vs Total Functions 


Data Points 
@ Historical Data 
Trend Lines 
..-. History Trend (mean) 
P=0,75 
y= 0.021601 .2482 
— History +160 
Filter 
Platform = Business Mission Critical 
Application = Financial Transactions 
Observations = 65 


_- 
a 
o 

= 

a 

<= 
c 
o 

= 
= 
rr 
hy 
w 

& 

ss 
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o 
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t 

= 
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a 
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Q 


Total Functions (UFP) (Log) 
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= A Point Estimate is 
never correct! 


= There is always 
potential variability in 
inputs. 

= But... 


26% of program managers 
do not accept risk 
assessment at all, not even 
“slightly”.* 

44% of risk ranges are 
intuitive judgments, without 
historical data or guided- 
survey.* 


*U.S. Aerospace Risk Analysis Survey, 
Hollis M. black, Ill, SCEA Estimator October 1999 
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Advanced Development - Total 


Government In-House Costs @ 35% 


dvanced Development - Contractor 


Program Management 
System Engineering 

Design Engineering 
Hardware Manufacturing (Prototype) 
Software Development 
Tooling and Test Equipment 
System Integration 

System Test and Evaluation 
Data 

Training 

Facilities 

Spares 


System Develop. & Demo. - Total 


Government In-House Costs @ 35% 


‘System Develop. & Demo. - Contractor 


Program Management 
System Engineering 

Design Engineering 
Hardware Manufacturing (Prototype) 
Software Development 
Tooling and Test Equipment 
System Integration 

System Test and Evaluation 
Data 

Training 

Facilities 

Spares 


Procurement - Total 


Government In-House Costs @ 25% 


Procurement - Contractor 


Program Management 
System Engineering 

Design Engineering 
Hardware Manufacturing 
Software Development 
Tooling and Test Equipment 
System Integration 

System Test and Evaluation 
Data 

Training 

Facilities 

Spares 

Operational Site Activation 


O&S Costs - Partial Total 


Fielding 

Sustainment (O&S) 
Spares and Repair Parts 
Personnel 


Low 
68,348 
17,720 
50,628 

2,448 
4,805 
4,526 


Uncertainty and Risk 
(Source Q. Redman 


Nominal 
97,132 
25,182 
71,950 

5,103 
7,731 
9,980 


24,150[ 25,421] 
14,064 14,805 


Low 
184,881 
47,932 
136,949 
2,448 
13,155 
16,093 


1,034 
1,880 
3,541 
1,013 
288 
627 
528 


Nominal 
273,557 
70,922 
202,635 
17,285 
17,547 
21,952 


85,865) 90,385 
18,753 19,740 


45 


Low 
2,149,874 
429,975 
1,719,899 
36,007 
18,003 
0 
1,497,878 


2,866 
6,368 
11,993 
3,432 
6,623 
2,123 
2,323 


Nominal 

3,162,081 
632,416 
2,529,665 
116,424 
98,129 
28,274 
1,576,714] 


77,994 82,099 


18,003 
18,003 
23,404 
3,601 
9,002 
0 
18,003 
0 


Low 
4,241,190 
747,220 
2,689,990 
747,220 
56,760 


83,576 
107,276 
144,698 

33,680 

30,769 

7,484 
142,703 
77,838 


Nominal 
5,173,716 
913,742 
3,289,472 
913,742 
56,760 


SEER 7 Characteristics of a Dysfunctional 
———— Software Project (Source: Mike Evans, et al.) 


Failure to Apply Essential Project Management 
Practices 57% 


Unwarranted Optimism and Unrealistic Management 
Expectations 41% 


Failure to Implement Effective Software Processes 
30% 

Premature Victory Declarations 20% 

Lack of Program Management Leadership 13% 
Untimely Decision-Making 8% 

Lack of Proactive Risk Management 3% 
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SEER Estimation Lessons Learned 


GA LOR A Tish 


The longer the development schedule the higher the 
risk of key personnel turnover 
EVM shouldn’t be used alone 
Other metrics are necessary to be kept in concert with the EVM 
metrics in order to keep a project on track 
= Measure early 
Before trouble 
Early milestones that are under-budget are not necessarily a good 
sign 
Programs that skimp on the early, upfront planning; requirements 
and design work, will most likely be in trouble later. 
= Follow known, proven development processes 


= Estimation, planning, tracking, controlling — then 
using the information to do better next time 


= Recognize that you have a problem 
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Engineering in a Non-Development Environment 


Kathy King and Thomas Zience 


Date/reference/classification 


BAE SYSTEMS 


Challenges in a Level of Effort Services Organization - 1 


@ CMMI Challenges 
— Inability to develop an 
Organizational Set of Standard 
Processes — Little commonality 
among contracts 


— Inability to establish a project’s 


Maintenance 


defined process - Customer Operations 
mandated processes, procedures 
and tools 
— Little or no engineering — Fixes 
to existing functional architecture Changes & 


New Features 
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Challenges in a Level of Effort Services Organization - 2 


Maintenance 


Operations 


Changes & 
New Features 


« Business Challenges 


— Fixed staffing levels with changing 
workload demands 


e Help desk call volume 
e Inherited legacy system problems 


e Service Level Agreement 
requirements 


— Inability to share best practices —- 
e Customer controlled processes 


e Lack of commonality among 
contracts 


— Environment challenges 
e 24-7 operation 
e Remote, local, pager support 
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Integrating Engineering Practices at the Top Level 


Process Control 
BAE 


Recruiting Planning & Forecasting 


Program Execution 
through: 


sjoeuoD 


Skilled Resource 
Utilization 


Bbulaaulbuy 
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Cc 
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SW Development 


¢ New 
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* Legacy 


Customer ¢ New features 


e Fixes 
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Skilled Resource Utilization Processes 


« Describe business activities 
¢ Five basic processes 
— Define the Work 
— Design the Staffing Solution 
— Fill the Positions 
— Define the Project Approach 
— Monitor and Control 
 SRU processes integrate the Project Management, Engineering, and 
Support Process Areas (PAs) 
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Interpreting Key Terms in the Model 


@ Project ¢ Funding period 


@ System/Product «————>  ¢ Staffing Solution 


« Product Components «—————-+ ¢ Staff Member with specific skills 


@ Functional Architecture «—————~ ¢ Service Architecture 


¢ Interface Requirements «————+ ¢ Requirements for Interfaces with 
Customer, other Teams, and 
Other Staff Members 


¢ Technical Data Package «———, ¢ Skills, Certifications, and 
Clearances 


Requirements Development 


Customer Requirements and Expectations 
— Staffing number with specified skill sets 
— Service Level Agreements (SLAs) 
— Services to be performed 
— Staffing Constraints 
Con ops and functional architecture 
— Description of services, relationships and interfaces 
— Environment - multiple contractors, shifts, travel 
@ Analysis of requirements 
— Necessary and sufficient? 
— Balance (cost and workload variability) 
@ High interaction with other PAs 
— Technical Solution 
— Validation 


Maintenance Operations 


Development 
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Requirements Management 


@ Requirements include 
— Customer requirements — staff numbers and skills, SLAs 
— Derived requirements — based on design of staffing solution, validation 
activities — may include non-staff resources 
@ Requirements traceability matrix maps requirements to individuals 
@ New or changed requirements for staff are analyzed using the 
Requirements Traceability matrix 
— Reassignment of existing staff 
— New hire requisitions (may change based upon experience) 


CLIN 001 | ~~ x ie .* ade a fa 
to J. Doe 


Position 
description Requisition Status 
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Technical Solution 


Design of staffing has two perspectives 
— Individual staff — skill, experience level 
— Total staffing solutions 
Design factors 
— Position to skill match — primary and backup 
— Positions may be mapped to multiple requirements 
— Shift schedules — overlap and rotation alternatives 
— Presence — on-site, remote, pager 
— Emergency procedures 
Non-staff resources may be part of design, e.g. tools 
When applied 
Initial 


— Based on validation results - workload variation, 
additional skills 


— Attrition 
— Perceived future trends 


® 


Service A — 
Shift 1 


ry 


® 


¢ ¢ 


Service B 
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Technical Solution (cont.) 


¢ Implementation involves acquisition of resources 
— Transfers 
— Hiring 
— Supplier Agreement Management applies - 
e Staff are considered product components 
e Non-staff resources, e.g. tools, pagers 
¢ Design constraints 
— LOE limitations 
— Contract specifications 


Service A — 
Shift 1 


® 
Me af a 


Service B 
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Product Integration 


¢ Product Integration strategy — staged integration of staff and other 
resources into customer environment 


— Over time 

— Across shifts 

— Across geographical locations 
« Readiness of products for integration — ability to bring staff on board 
Interface specifications — part of the communications plan 
« Assemble the product components — staff members 

— Orientation and security briefings 

— Mentor assigned 
« Evaluation of assembled product 

— Verification of skill level 


— Non-staff support resources (e.g. pagers, or phone 
call transfers) 
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Verification 


¢ Verification - Staffing satisfies contract requirements — what the 
customer states is required 


— Staffing levels 
— Cost 
+ SLAS 
« Defect analysis and resolution 
— Upgrading staff skills 
— Reassignment or scheduling 
@ Peer reviews 
— Mostly performed on plans 
— Internal transfers — assignments panel 
— Staffing candidates — candidate evaluation form 
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Validation 


¢ Satisfaction of customer expectations — what we determine is needed 
beyond customer stated requirements 
¢ Probably the most important Process Area in a services environment 
— Ultimate project success — ability to retain staff 
— Growth in business with customer 
@ Are the contract requirements sufficient for success? 
— Can SLAs be met with current staffing solution? 
— Is staffing design sustainable? 
— Plan for Award/Incentive fee 
@ Performed throughout the contract period — at least monthly 
@ May result in 
— Redesigning the staffing solution 
— Incorporating non-staff resources into design 
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Mapping Our Process to CMMI 


Define the Work 
Design the Staff 


Define Project 
6 Approach 


Fill the Position 


Monitor and 
Control 
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Summary 


« Engineering Process Areas add rigor to designing and managing staff 
solutions in an LOE Services environment 
¢ Benefits of Integrating the Engineering and Project Management 
Process Areas above the line 
— Relieved confusion with similarly name below the line processes 
— Led to a business driven process architecture 


— Opened new opportunities for business driven improvements beyond 
project teams — recruiting, finance and planning 


« Engineering Process Areas with the greatest impact to business 
— Technical Solution — ability to design complex staffing solutions 
— Validation — evaluate and improve staffing solutions 
« Appraisal challenges 
— Quality Assurance is performed on SRU processes 
— Appraisal requires Process Area goal satisfaction 
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Contact Information 


Kathy King — BAE Systems Information 
Technology 
—kathy.king@baesystems.com 

Thomas Zience - BAE Systems Information 
Technology 


— thomas.zience@baesystems.com 
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Topics 


= Overview Of Our Organization And Process 
Improvement History 


= Organizational Scope 
= Different Views 
= Restricted Scope For Materiel 
= Shared Vision 
= Our Definition 
= Shared Vision Flow-down 
= Organizational Process Areas 
= Integrating Them 
= Supporting Infrastructure 
= Checking The Extent Of Integration 
= Continuing To Grow 
= Summary Of What We Learned NORTHROP GRUMMAN 
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Corporate Overview 


= Comprehensive business 
portfolio to address the 
defense and government 
markets 
— Systems integration 
— Military aircraft 
: — Unmanned aerial vehicles 
— C4ISR 
— Defense electronics 
| — Information technology and 
networks 
— Naval shipbuilding 
| — Space and missile defense 
| 


= 2005 sales of $30.7 billion 
= 125,000 people; 50 states; 
25 international countries 


= Headquartered in Los 
Angeles, CA 
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gu esa 
77% 


IMM AR 


Ef as las Fae 


Northrop Grumman Integrated Systems 


E-2C/AHE 


Re NATO Ags  ADYVANS=0 2 


™’ ¢ PTS. 
~ 
a» 


EA-186 ICAP III 


Premier Aerospace and Defense Systems 
Integration Enterprise 
NORTHROP GRUMMAN 
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Ww AGS&BMS Major Sites 


/ 


Level 5 CMMI-SE/SW/IPPD/SS 6/06 
Level 5 CMMI-SE/SW 4/05 
ISO 9001:2000/TickIT 9/04 


Pp Lf O/C ig (a e/a E774677" 1 
5 a 
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Process Improvement History 


SM 
SW CMMS™ cw 


om 


Engineering Broader 
Focus Focus 


SE CMMS™ 


ISO 9001 
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Expand Your Organizational Scope 


=" Benefits seen in software engineering were expanded 
into systems engineering 


_ = Why not expand these benefits into other areas? 


= Expectation was that expanded scope would yield 
progressively greater process awareness and benefits 
= Initial expansion was 
= The rest of engineering beyond systems & software 
= Materiel - to bring in supplier sourcing practices 
= Program Management - to bring in IPPD practices 
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Organizational Scope 


=" When the model was SE/SW - the organization was 
“engineering” 


= When non-engineering departments are included, 
what does the term “organization” mean? 


= Be careful in using the term “organization” - the CMMI 
has requirements for the “organization” 
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AGS & BM Systems 


Business Unit (Sector VP) 


im Ulaterd(e)arsl| ‘ 
baci acto Met=) avi (er) 
‘ => Site Managers Nomeroome Support Services 


Human Resources Melbourne, FL Executive 
& Administration Support 


Engineering, 
Logistics & 
Technology 


—_ — Manufacturing 
Communications Programs Operations 


Quality 
Program Operations 
Management 


Engineering Disciplines 
NORTHROP GRUMMAN 


Material 
Management 


Quality Systems & 
Process Excellence 


Information 


Technology 


Reports To Management 


9 Above The Business Unit 
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Organizational Scope - Different Views 


= What is your organizational scope? 
= How do you want it portrayed? 


Organizational view (all departments are equal - maybe) 


Program 
Engineering Management 


Project view Program 
(PM manages the job) Management 
ee 


NORTHROP GRUMMAN 
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Restricted Scope for Materiel 


= The Business Unit is the organization being appraised 


= Materiel is INSCOPE 


= Materiel is considered part of organization for project- 
type process areas (mostly SAM, MA, ISM, PI and IT) 


= Generic practices (e.g., defined process, plans, 
resources, responsibilities, training, manage 
configurations, relevant stakeholders, monitor and 
control process, review status, objectively evaluate 
adherence) 


= Materiel is OUT OF SCOPE 


= Materiel is not considered part of organization for 
organizational process areas (OPF, OPD, OT, OEl) 


= Materiel is not considered part of organization for 
“organizational” generic practices (e.g. organizational 
policy, collect improvement information) * (ee 
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Shared Vision 


»4* It’s the “organization’s” shared vision - not just 
a *\ engineering 


= How “high” up in the organization do you go? 
= How do you get it flowed down? 


= How low in the organization do you flow the shared 
vision? 


NORTHROP GRUMMAN 
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Shared Vision - Our Definition 


Our “shared vision” consists of our business 
objectives with an associated set of expected 
values and behaviors. 


= The business objectives are defined at a high 
level for corporation and are flowed down and 
decomposed for the sector, business area, 
projects/teams and individuals 


= The set of values and behaviors are defined at 
the corporate level and flowed down “as is” to 
the sector, the business level and ultimately to 
every individual at Northrop Grumman 


NORTHROP GRUMMAN 
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MORTHIOE CRUPPEANM 
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ETHICS 
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TAKING THES ite ROAD 
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ared Vision 


Gur vision Is to be the most trusted provider af 
systems and tochaologjes that ensure the security and 


freedom of our aation and Lts allies. As the techaotogy 


leader, we will Geflae the future of daferse—troen 
undersea to outer space, and ia cybempace, 


We wo 
B Conduct vee witty Li erat live 
—~a 


1B Datrrer supecior prouran pectormuexe 


© ooter an interred ecerorenent of innemstion 
callsbortion, seid nt 


Taso doing, Nocthrop Griumian will become our 


customers’ partiier of choice, our industry's employer 
of choice, and our starcholdery inwstment of choice, 


Ont 


Live the Company Values 
WM a Nowe the adios Veted on the tase 
ot cue test go—Chualty Sattutes oa, Laaeaesbips 


ewoeize menting ny do. Vin wa maryeim vin 
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Communicate 
ee 
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by the Following Values. They dercrthe our comparyy at yee 
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PEOPLE... 
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Nultebiabs and tekes. Chir continued sie require isto» 
provide the education and developrrant rmeded to help our 
people grow, We are committed to operieme anid trast tn all 
telaticembipe 


WE aecarp our SUPPLIERS as 
BSSENTIAL TEAM MEMBERS... 


We owe our suppliers the sume type of respect that wer 
hese to carr cuutomrs. Our suppliers daserve fair ars 
(quttable treat mere car ageremmres and honest feedback 
fon performance We conuider our mupplisrs ends tn, 
Fandocting all pects of our bssiewen, 
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Shared Vision Flow-Down 


Corporate 


IS Sector 


AGS 
Business 
Area 


Project & 
Functional 
Teams 


Team 
Members 
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Values/Behaviors 


Se sector Goals 


Unchanging 
Values & 
Behaviors 


Business Area Goals 
for Functional Teams 


e Monitoring Process (PMP) 


Performance Monitoring Process (PMP) 


=" Yearly defined tasks, projects, & goals identified (including 
requirements, schedule, cost, quality) - evaluated at least three 
times throughout the year 

_= Ethical conduct evaluated - Adherence to company standards of 

conduct; behavior and decisions are consistent with Northrop 
Grumman Values 

= Work style characteristics evaluated including 
= Collaboration/Teamwork 
» Process Improvement 
» Interpersonal Skills 
=» Leadership (Managers Only) 

» Leadership competency (for managers) evaluated, including 
" Acts Ethically 
=» Builds Teams and Relationships 
= Treats People with Respect, Including Customers, Suppliers, and 

Competitors 

= Collaborates Effectively 
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Organizational Process Areas 


= You need to map the CMMIS™ organizational 
requirements onto the organization you have 
identified as within scope 


= You need to determine if and to what degree different 
parts of the “organization” need to be integrated 


NORTHROP GRUMMAN 
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Organizational Process Areas 


= Is there an integrated approach to the CMMIS™ 
organizational process areas? 


= OPF - Process improvement activities (are we 
leveraging improvements across the organization?) 

= OPD - Process assets (common repository?) 

= OT - Training (one training plan or multiple?) 

= CEI - Integrated Work Environment (is it really 
integrated’?) 

= OPP - Performance Baselines (how do they “flow up” or 
“combine”?) 

"= OID - Improvements & Innovations (how are they 
selected across the “organization’”) 

= GP 2.1 Organizational Policy (does it cover the whole 
“organization”) 

= GP 3.2 Collect Improvement Information (is it shared 


across the “organization”) ee an aC Te 
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Integrating the Organizational Processes 


Corporate 
Sector 
Business Area 
Executive Management 


¢ Corporate goals 


¢ Sector goals ¢ Status reporting 
¢ Business Area goals 


* Two OPG Co-Chairpersons 
(Process Excellence & Engineering) 
e Representatives from each Dept. 


¢ Direction, Guidance ( ; ¢ Improvement proposals 
e Resources ¢ Process performance 
* Goals, Objectives status reporting 


Departments 
& Directorates : 

Materiel EL&T 

Manufacturing SE&T 

Product Quality Software 

Business Mgmt Test & Eval 
Multi-disciplinary teams Program Mgmt Vehicle Eng 
empowered to evaluate and Process Management Business Developmt Avionics 
continuously improve Teams Command Media Logistics 
organizational processes Human Resources Eng Project Mgmt 

Sites Lean / Six Sigma 

Programs 


NORTHROP GRUMMAN 
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Is Your “Organization” Really I ntegrated? 


= Are there any “feel good” metrics you can check? 


NORTHROP GRUMMAN 
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Procedure Distribution 


Admin @BD O Contracts OPM m ELT @ Finance 
w@ HR O Facilities m Material w Quality MO Prop Mgt Enterprise 
fe | Admin 
Enterprise WNW 
Quality 9% 
1% BD Contracts 
SN 3% j 6% 
\S ) 
Material 
13% 
PM 
19% 
Facilities 
1% 
A% 
Finance 
1% 
ELT  - 
40% 
Procedure Data 
Bus Property 
Admin Dey Contracts PM ELT Finance HR Facilities Material Quality ©=Managerent Enterprise Total 
4 6 11) 35) 75 2 ff 2 25 2 1 16 186 
NORTHROP GRUMMAN 
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Artifact Distribution - 


Admin @ BD 11 Contracts _ 1 PM m@ ELT @ Finance 
HR OFacilities MM Material H Quality O Prop G@ Enterprise 
Mot 
Prop . 
Migt ou BD 
O% © Enterprise . 5 
| \ aie | [| 1% 
Quality + [ Contracts 
1% % Xe # A% 
Material — 
12% PM 
eeeus / 35% 
Facilities _ Z 
O% 
HR © 
2% 
Finance 
2% 
| 
ELT 
| 31% 
Artifact Data 
Bus Property 
Admin Dey Contracts PM ELT Finance HR Facilities Material Quality Management Enterprise Total 
ee | ers | 129 1157 1030 79 8650 14 397 19 3 374 3302 
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Growing the Capability 


What happens after Level 5... 


Optimized _ 
Processes _ 


Projects 


Departments. 


... You Keep Improving 
NORTHROP GRUMMAN 
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Summary 


= Don’t be afraid to expand CMMIS™ usage outside of 
engineering 

= It may be easier to satisfy CMMIS™ practices as you 
move outside of engineering 


= Many non-engineering departments have excellent 
process capability - you can learn from them 


= Executive management will likely be strong 
supporters of expand organizational scope 


= Practitioners (especially project personnel) will likely 
be strong supporters - they recognize that many 
departments are needed to successfully develop 
products 


= You may find redundant processes that can be 
streamlined and interfaces that can be improved 
NORTHROP GRUMMAN 
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QUESTIONS 


123 


Joseph V. Vandeville 
Northrop Grumman Corporation 
(321) 951-5287 
joseph.vandeville@ngc.com 
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Background 


*4 = CMMI v1.2 represents a major improvement in the CMMI, 
— responding to thousands of community change requests 


jr | = The new model is simpler and more straightforward in some 
~ 7). ~~ Ways, but more complex in others. 


= Current users of CMMI v1.1 should carefully consider strategies 
in transitioning to the new model 
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= Model changes and strategies for transition 


WV = Model architecture 

Q = Supplier management 
_ = IPPD (even for those who don’t currently use it!) 
y = Process deployment 


= Appraisal changes and strategies for transition 
= ADS 


= What does it all mean? 


SM SCAMPI, SCAMPI Lead Appraiser, and SEI are service marks of Carnegie Mellon University. 
® Capability Maturity Model Integration and CMMI are registered in the U.S. Patent & Trademark Office. 
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Key References 


= http:/Awww.sei.cmu.edu/cmmi/adoption/cmmiv12-changes.html 
=» “CMMI v1.2,” David Phillips, SEI (Monday tutorial) 


» “CMMI V1.2 Overview,” David Phillips, SEI (Wednesday presentation) 
» “SCAMPI 1.2 Updates,” Dr. Jack Ferguson, SEI (Wednesday presentation) 


THE ARCHITECT COLUMNS ¥ cmMMI IN FO 


CMMI-DE¥Y Version 1.2: What Else Has Changed? 
Home Searclh ContactUs SiteMap What's New MIKE PHILLIPS 


Carnegie Mellon 
Software Engineering Institute 


LINES CMMI for Development (CMMI-DEYV), Version1.2 includes improvements to all parts of the CMMI 


Imi rovin ret Management Engineering Acquisition Work with Us eyed Publications Product Suite in response to issues that have emerged in practice, The changes focus on 
managed and Services improving the quality of CMMI products and the consistency of how they are applied. 
practices 


The CMMI V1.2 Product Suite contains the following items: 


e CMMI for Development (CMMI-DEV), V1.2 model 
Issue © Appraisal Requirements for CMMI (ARC), V1.2 

@ SCAMPI 4 V1.2: Method Definition Document 
Introduction to CMMI V1.2 course 


MANAGEMENT Comparison of Version 1.2 to Version 1.1 


© CMMI Main To make it easier for you to see content changes to model ae ® 
Page material between CMMI v1.1 and CMMI for Development 


J 
© What Is CMMI? v1.2, the following comparison files are available. These 
comparison files highlight the differences between the Cmmi 


. 
Models following elements: as 


© Adoption 
H 
© Tpainin Process areas 


Events, & 
Forums 


Se eer odies of knowledge that are essential for 
development and maintenance but that have been addressed separately in the past, such as 
software engineering, systems engineering, and others, CMMI V1.2 embodies the concept of 
CMMI constellations, in which a set of core components can be augmented by additional 
materials. CMMI-DEY is the first such constellation. Currently there are plans for three 
constellations supported by the V1.2 model framework: development, acquisition, and services. 


What are the major changes to the model document? 


In earlier columns, I described most of the intended improvements that would be considered for 
the CMMI V1.2 model. These changes included eliminating three process areas—ISM, OEI, and 
IT—while preserving coverage of these important practices. This change reduced the size of the 


Nh Oa hilt 


Generic practices 


° Performance ay 
hr gi Glossary definitions 


Results 
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Model architecture changes 


*4 = A“constellation” approach was adopted for the family of CMMI 
—_/ models to be developed 


Ay =" CMMI for Development (CMMI-DEV) takes the place of the old 
, CMMI v1.1 


= CMMI for Services (CMMI-SER) addresses services 


= CMMI for Acquisition ACQ (acquisition) addresses the acquisition 
of products and services 


= Added hardware amplifications 
= Eliminated advanced practices (little strategic impact) 


= Eliminated common features (little strategic impact) 


ee 
ty 
eae 


Strategy Implications 

¢ Consider expanding the scope of your improvements to include hardware, services, 
acquisition 

¢ New disciplines may struggle with basic adoption issues (e.g., Level 1 to 2) 


NORTHROP GRUMMAN 
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Elimination of 
Integrated Supplier Management 


V1.1 
a . 
— Supplier Agreement Management 
a” SP 1.1 Determine the type of acquisition for 
each product or product component to be 
a acquired. 
ne 
_ SP 1.2 Select suppliers based on an evaluation 
of their ability to meet the specified 
requirements and established criteria. 
SP 1.3 Establish and maintain formal 


agreements with the supplier. 


SP 2.2 


Supplier agreement. 


Perform activities with the supplier as 
specified in the supplier agreement. 


—integrated-Supptier-Managenrent 


SP 2.1 


SP 2.2 


SP 2.3 


Monitor and analyze selected processes 
used by the supplier. 


For custom-made products, evaluate 
selected supplier work products. 


Ensure that the supplier agreement is 
satisfied before accepting the acquired 
product. 


SP 2.4 Transition the acquired products from the 


supplier to the project. 


V1.2 


Supplier Agreement Management 


SP 1.1 


SF 12 


SP 1.3 


SP 2.1 


SP 2.2 


SP 2.3 


SP 2.4 


SP 2.5 


Determine the type of acquisition for 
each product or product component to 
be acquired. 

Select suppliers based on an evaluation 
of their ability to meet the specified 
requirements and established criteria. 


Establish and maintain formal 
agreements with the supplier. 


Perform activities with the supplier as 
specified in the supplier agreement. 


Select, monitor, and analyze processes 
used by the supplier. 


Select and evaluate, work products from 


“the supplier of custom-made products. 
Ensure that the supplier agreement is 
satisfied before accepting the acquired 
product. 


Transition the acquired products from the 
supplier to the project. 
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Strategies for transitioning to the new 
Supplier Agreement Management 


At the organizational level: At the project level: 


_ = #\ = Classify the types of suppliers used within the . 
organization 
7 = True subcontractors (who follow their own 
- processes to produce custom-made products) 
= Integrated suppliers (who follow our 
processes) a 

= COTS vendors 
= In-house suppliers 
= Customers (GFE) 


= For each type, determine appropriate methods 
(and responsibilities) for projects to: 


= Evaluate and select suppliers (sp 1.2) 
= Establish formal agreements (SP 1.3) 
= Work with the suppliers (SP 2.1) 


= Select key processes; monitor and analyze 
them (SP 2.2) 


= Select key work products; evaluate them 
(SP 2.3) 


= =» Ensure the agreement is satisfied (SP 2.4) 
= Transition the acquired product (sp 2.5) 


= Define acceptable variations of these methods 


Determine the type of 
acquisition for each 
product or product 
component to be 
acquired 


Define the methods (and 
variations) to be used 
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Sample Organizational Guidance 


Evaluate and select 
suppliers 


RFP, process evaluation 


Establish formal agreements | Contract 


Work with the suppliers 


Select key processes 


Monitor and analyze 


Select key work products 


Evaluate them 
Ensure the agreement is 
satisfied 


Transition the acquired 
product 


Monthly status meetings, 
milestone reviews 


Req definition, testing 


Req Review, quantitative 
analysis of defects 


Specification, test plans 


Peer review 
Witness sub test 


Tape transfer 


COTS Vendors 


Use pre-approved vendor 


Purchase Requisition 


NA 


Inspection 
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| PPD changes 


Project Management 

Project Planning 

Project Monitoring and Control 

Supplier Agreement Management 
1 * 


Kk 


Integrated Project Management (for IPPD*) 
Risk Management 
Quantitative Project Management 


Engineering 

Requirements Development 
Requirements Management 
Technical Solution 

Product Integration 
Verification 

Validation 


*IPPD extension in v1.1 
**Supplier Sourcing extension in v1.1 


Support 

Measurement and Analysis 

Process and Product Quality Assurance 
Configuration Management 


Causal Analysis and Resolution 
Decision Analysis and Resolution 


Process Management 

Organizational Process Focus 
Organizational Process Definition 
Organizational Training 

Organizational Process Performance 
Organizational Innovation and Deployment 


IPPD Changes 
¢ Removed OEI, added to OPD 
« Removed IT, added to IPM 
¢ Moved “Enable IPPD Management” to OPD 
¢ Moved “Apply IPPD Principles” to IPM 
¢ Revised IPPD material to be consistent with 


the other model material 
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|PPD Extension 


ace. 
ss Sa 


10 


Organizational Process Definition +IPPD 

SG 2 Organizational rules and 
guidelines, which govern the 
operation of integrated teams, 
are provided. 

SP 2.1 Establish and maintain 
empowerment mechanisms to 
enable timely decision making. 

SP 2.2 Establish and maintain 
organizational rules and guidelines 
for structuring and forming 
integrated teams. 

SP 2.3 Establish and maintain 
organizational guidelines to help 
team members balance their team 
and home organization 
responsibilities. 


Integrated Project Management +IPPD 

SG3_ The project is managed using 
IPPD principles. 

SP 3.1 Establish and maintain a shared 
vision for the project. 

SP 3.2 Establish and maintain the 
integrated team structure for the 
project. 

SP 3.3 Allocate requirements, 
responsibilities, tasks, and 
interfaces to teams in the integrated 
team structure. 

SP 3.4 Establish and maintain integrated 
teams in the structure. 

SP 3.5 Ensure collaboration among 
interfacing teams. 


Strategy Implications 


¢ The resulting IPPD extension reflects good management 
¢ These practices could be adopted on projects not organized 


around Integrated Product/Process Teams 


¢ Organizations may wish to develop IPPD rules for 


deployment to some projects 
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Additions to the non-I PPD version of the model 


Organizational Process Definition Integrated Project Management 
"gq «9? 1.6 Establish and maintain work SP 1.3 Establish and maintain the project's 
Fi environment standards work environment based on the 
rf organization's work environment 
> standards. 


Strategy Implications 


¢ The environment (tools) is now considered part of the 
organization’s standard process, and must be tailored by 


the projects NORTHROP GRUMMAN 
a 
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Process deployment changes 


Organizational Process Focus 


SG3_ The organizational process 
assets are deployed across the 


organization and process-related 
experiences are incorporated into 


the organizational process 
assets. 

SP 3.1 Deploy organizational process 
assets across the organization. 


SP 3.2 Deploy the organization’s set of 
standard processes on projects at 
their startup and deploy changes to 
them as appropriate throughout the 
projects’ lives. 

SP 3.3 Monitor the implementation of the 
organization’s set of standard 
processes and use of process 
assets on all projects. 


SP 3.4 Incorporate process-related work 
products, measures, and 
improvement information derived 
from planning and performing the 
process into the organizational 
process assets. 


Integrated Project Management 


SP 3.1 Establish and maintain the project's 
defined process from project startup 
through the life of the project. 


Strategy Implications 
e The organization must define how 
changes to the standard processes will be 
redeployed to projects 
— This may suggest re-architecting the 
structure of the standard process, i.e., 
parts that change over time and parts 
that don’t 
— Impacts on the tailoring process 
e ALL projects should be using the standard 
processes and process assets 
— Gives appraisal teams the right to look 
at other than” focus projects” — should 
deter “cherry picking” 
— The monitoring of process assets use 


may require technology or procedures 
for the Process Asset Library y 
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Appraisal changes 


=» Rating changes 
aad = Process areas outside of the model 
‘ scope are rated as Out of Scope 


= Process areas that have insufficient 

rT data to be rated are rated as Not 

a Rated 

| = Process areas in the model scope, 
but outside the organizational scope 
are rated as Not Applicable 


= The only process area that can be 
Not Applicable is SAM (determined 
by the appraisal team) 


= Appraisals can now include better 
sampling approaches: 


= Focus projects — all PAs 
=» Non-focus projects — some PAs 
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= The Appraisal Disclosure Statement 
(ADS) now requires: 


= Organizational sampling criteria and 
decisions (e.g., projects included, 
projects excluded, percentage of 
organization represented) 


= Basis for maturity/capability level 4 
and 5 appraisal results must be 
mapped to quality and process- 
performance objectives 


Strategy Implications 
e An organization that does not implement 
all processes can not get a ML rating 
e May require re-write of contractual 
requirements, which means educating 
the customer on model applicability 
¢ Non-focus projects allow better sampling 
approaches 
e Although the new ADS does not change 
the sampling process, it provides greater 
visibility 
— How will customers use this insight? 


pe 


Copyright 2005 Northrop Grumman Corporation 


Appraisal Sampling Strategies 


Organizational Unit 


=" How many focus projects? Non-focus? 


=" Select focus projects by size? Importance? Typicality? 

=" How many process areas to sample in the non-focus projects? 
= Which process areas to sample? 

= Sample common process areas? 


= Who sets sampling — Organizational Unit? Lead Appraiser? 


NORTHROP GRUMMAN 
14 


Copyright 2005 Northrop Grumman Corporation 


Northrop Grumman Mission Systems 
Process Deployment Approach 


*4 = All projects perform process tailoring during proposals 
eal 4 = Without a defined process, it is impossible to bid accurately 


= Templates are used to generate plans consistent with the project’s 
defined process* 


= All projects undergo a CMMI-based evidence review within the 
first 90 days 


= Ensures that projects are implementing proven processes 


= All projects participate in our SCAMPI A appraisals 
= Ensures that all projects benefit from mature processes 
= Send clear message that CMMI has value** 


*” Project Implementation Strategies in the CMMI ,” R. Hefner, CMMI 
Technology Conference and User Group, 2006 (Wednesday PM) 


**” Sustaining CMMI Compliance,“ R. Hefner, CMMI Technology 


Conference and User Group, 2006 (Thursday AM) NORTHROP GRUMMAN 
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*4 = CMMI v1.2 represents a major improvement in the CMMI, 
me responding to thousands of community change requests 


a | = The new model is simpler and more straightforward in some 
 -_ ways, but more complex in others. 


= Current users of CMMI v1.1 should carefully consider strategies 
in transitioning to the new model 


NORTHROP GRUMMAN 
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CMMI? Version 1.2 SAM+ISM 


NDIA CMMI Technology Conference 
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® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. 
Thanks to Gary Wolf and D’Ann Hunt from Raytheon; Denise Cattan of SPIRULA; Rick 
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Carnegie Mellon 
Software Engineering Institute 


Major Themes 


Reduce complexity & size 
Increase coverage 


Increase confidence in appraisal results 
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Carnegie Mellon 
Software Engineering Institute 


Reduced Model Complexity & Size 


Eliminated the concepts of advanced practices and 
common features 


Incorporated ISM into SAM; eliminated Supplier Sourcing 
(SS) addition 


Consolidated and simplified the IPPD material; eliminated 
two process areas (OEI, IT) 


Consolidated all definitions in the glossary 
Adopted a single book approach 


Reduced report size 15% 
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Historical Perspective 
Software Subcontract Management (CMM) became 
Supplier Agreement Management (CMMI V1.0) 


¢ “Not applicable” designation in appraisals reduced from 
56% to 20% 


e Emphasis on COTS added due to historical difficulties 


V1.02d added acquisition flavors with 3 levels of supplier 
attention (SSM, ISM, QSM) 


V1.1 consolidated these into “Supplier Sourcing” PA, ISM 
¢ Some SAM subpractices become ISM specific practices 


e Multiple CRs argue for consolidation 


=> Carnegie Mellon 


= Software Engineering Institute 


CMMI Model Combinations 
V1.1 V 1.2 


. 
Organizational Goal 
(OPD) 
Project Goal (IPM) 
Le 
Related 


Examples 


——_—— ap 
Supplier 
Sourcing 


ntegrated Product anc 
Process Development 
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Consolidation Themes 


Eliminate SS distinction to reduce complexity 
Reduce PA number to simplify appraisals 


Maintain ISM theme of greater insight into supplier's 
development activities to reduce product development risk 


Reposition COTS emphasis to “make-buy” analysis in TS 
Avoid “raising the bar” for SAM 


iff 
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Supplier Agreement Management 


Specific Goal Specific Practice 
Establish Supplier 1.1 - Determine Acquisition Type 
Agreements 1.2 - Select Suppliers 


1.3 - Establish Supplier Agreements 


Satisfy Supplier 2.1 — Execute the Supplier Agreement 
Agreements 2.2 — Monitor Selected Supplier 
Processes 
2.3 — Evaluate Selected Supplier Work 
Products 


2.4 — Accept the Acquired Product 
2.5 — Transition Products 


v1.1 SP2.1 “Review COTS Products,” was eliminated. “Identify 
candidate COTS products that satisfy requirements” is a new 
subpractice under the Technical Solutions Process Area SP1.1, 
“Develop Alternative Solutions and Selection Criteria.” 
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Key informative material 


“The content of the agreement should specify the reviews, 
monitoring, evaluations, and acceptance tests to be 
performed, if such activities are appropriate to the 
acquisition or product being acquired.” (SP1.3) 


“For most vendor agreements where a product is not 
being developed or for smaller, less critical components, 
the selection process may determine that monitoring is not 
appropriate.” (SP2.2) 


“The scope of this specific practice is limited to suppliers 
providing the roject with custom-made products, 
particularly those that present some risk to the program 
due to complexity or criticality.” (SP2.3) 


= Carnegie Mellon 
= Software Engineering Institute 


Sample Organizational Guidance 


COTS Vendors 


Use pre-approved vendor 
Purchase Requisition 


Work with the suppliers 


Select key processes 
Monitor and analyze 


Transition the acquired 
product 
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Small Project Challenges of CMMI in 
Large Organizations 


6 Annual CMMI ® Technology 
Conference and User Group 


Hyatt Regency Tech Center, Denver, CO 
November 16, 2006 


Mr. John Bethmann 
Principal Systems Engineer 


Ms. Ruth Wuenschel 
Principal Systems Engineer 


Mr. Corey Norris 
Manager Systems Development Process 


Agenda 


e |ntroductions 

¢ Overview of Our Company 

¢ Describe the Challenges we faced 

e Describe how we categorized projects 

e Show how we tried to “Address the 
challenges” 


¢ Closing comments 


Our Company - Concurrent 
Technologies Corporation (C7C) 


e An independent, nonprofit, applied research and 
development professional services organization 


e Staff of 1,500+ professionals 

e More than 35 locations 

e Approximately 900,000 sq. ft., including laboratories 
and demonstration facilities 

e Federally compliant contractor 

¢ Concurrently certified to 1SO 9001/ 14001 in August, 1998 

e Assessed at Maturity Level 3 of the Capability Maturity 
Model Integration Systems Engineering/Software 
Engineering (CMMI-SE/SW®), Version 1.1 in March, 2003 
for systems and software engineering activities 


¢ Certified to AS9100 in April, 2005 for aerospace activities 


CTC Locations 
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CTC location ° 


Annapolis J unction, MD « Bremerton, WA « Charleston, SC* « Columbia, SC « Crystal City, VA «+ Dayton, OH 
Fayetteville, NC + Fort Leonard Wood, MO « Greenville, SC «+ Hamilton, ON, Canada « Harrisburg, PA 

Jacksonville, FL « Johnstown, PA « Largo, FL « Panama City, FL + Pensacola, FL* « Pittsburgh, PA* « San Diego, CA 
Stuttgart, Germany « Washington, DC* 


* C7TClocation and C7C on-site location 


CTC on-site location 

Albany, GA « Alexandria, VA « Camp Lejeune, NC « Fort Bragg, NC ¢ Fort Dix, NJ + Indiana, PA « Lackland AFB, TX 
Morgantown, WV + Mountain Home AFB, ID « New Carrollton, MD « Norfolk, VA « Philadelphia, PA « Shalimar, FL 
Tampa, FL + Tobyhanna, PA ¢ Virginia Beach, VA « Warner Robins, GA « Wright Patterson AFB, OH 


Our Growth Profile 


I started here | 


Staff Make-up 


1% 3% 


& Doctoral 

@ Masters 

@ Bachelors 
G Associates 
@ Technicians 


Staff Disciplines 


Electrical & Mechanical Engineering 14% 


Computer Science & Mathematics 21% 
Communications Technolog 23% 


Environmental & Chemical Engineering 9% 
Metallurgical & Materials Engineering 3% 
Engineering & Science 9% 


Business & Others 21% 


Selected Services and Capabilities 


¢ Advanced Coatings and Coatings ¢ Intelligence Analysis 
Removal ¢ Logistics Optimization 

e Advanced Distributed Learning e Management Systems 

e Advanced Materials and Processes ¢ Manufacturing | mprovement 

e Automation, Controls, and * Modeling and Simulation 
Data Acquisition ¢ Supply Chain Integration 

¢ Command and Control Systems ¢ Sustainability 

* Communications and Outreach * Systems Design and Analysis 


¢ Corrosion Prevention and Control = oystems tntegraton 


¢ Systems/Software Engineering 
¢ Environmental Technology 


Demonstration and Validation * Technology Management 
¢ Visualization 


Fuel Cell Test and Evaluation 


¢ Information Assurance 


Challenges We Face 
CTC Project make-up 


e Number of People: 1500 So What? 
¢ Number of Projects: 230 


° Average Project Size 5 FTEs e Although it looks Large 
¢ Typical Large Program: 35 FTEs it’S small 
¢ Typical Small Program: 1 FTE e Processes need to tailor to 
projects of varying size 
e We realized this impacts 
how we do business 


Challenges We Face 
Multiple Disciplines 


¢ Disciplines 
— Engineering suideicand 
_: Seshees e Broad types of projects 
~ Construction ¢ Intelligent staff with strong 
Management opinions 
— Communications ¢ Not practical to 
— Education and Training implement one size fits 
— Business (program all 
management and other) = 
- Variety of tools and e We realized this impacts 
templates required to how we do business 
support 


e Different terminology used in 
each field 


Challenges We Face 


Multiple Standards 

- 1SO 9001 So What? 

* AS 9100 e Underestimated effort 

¢ ISO 14000 required to train staff 

* VPP - Voluntary Protection . Large effort required to 
Program manage and maintain 

¢ CMMI - across the ¢ Not practical to 
company implement as multiple 

quality systems 


¢ We realized this impacts 
how we do business 


Current Procedure Presentation 


e Sequential Listing based on old |SO 9001:1996 
sections 


e Additions for each new standard 
e Need to improve usability of the system 


e Numbering scheme not structured to handle new 
procedures added over time 


How we categorized projects 


Quality Management System Environmental Health and 
Safety (QMS/EHS) 
— Quality Planning Checklist asks 
- “Is the project designated as a Systems 
Development project?” 
— Systems Development QA Lead determines If 
project is a system development or service. 
— Below the Threshold - 2100 hours 
— Projects are of two distinct types: 


— Service Projects 
— Systems Development Projects 


What happened 


e System definition was misunderstood by staff and 
management 


— Only for software types 
e Projects mixed services and systems 
— Lacked guidance on how to decide 


— People reverted to what they knew to get 
things done 


e Created auditing confusion for staff 


Addressing the Challenges 
What is the starting point 


e Analysis of internal records showed we have a 
variety of types and sizes of projects 


e Took a step back and said “What is C7C 


concerned with?” - Quality product that meets 
or exceeds the Clients needs. 


—- Within Budget 
—- Controlling Risk 


e Reviewed Technical Management Literature on 
Project Classifications 


e Decided to implement a Project Classification 
Framework based on size and risk 


Basis for Framework 


e Technical Management research conducted by 
Wesley J Howe School of Technology 
Management, Stevens Institute of Technology 

—- Dr. Aaron |. Shenhar 

e Ohio State ClO Office 


— http://oit.osu.edu/projmanage/PDF_files/OSUP 
MErmwk_ 1.00. pdf 


Sizing Matrix 


Project Classification - Sizing Matrix 


Project Class Lower Limit Upper Limit 


Risk Matrix 


=a eg ae 
= 


Team — | ne Gama 14 


# haa =a 
Groups 


Technology / | Expert a New to CTC | Cutting Edge 
Technique / / Rocket 
Process Science 


Complexity Well Known Multiple Unknown or 
defined no | some approaches | vaguely 
problems | problems defined 


Political ma Medium Large Impact | Very Large 
Profile / | mpact | mpact 
I mpact 


Project Classification - Risk 
Adjustment 


e Risk adjustment 
- Oto 10 > no affect 
— 11 to 13 > Increase 1 level 
- 14to 15 > Increase 2 levels 
e Size Matrix + Risk adjustment = Project Class 


Phases of the “System” 


e Proposal Development 
¢ Contract Award 

e Planning 

e Execution 

¢ Closeout 


Configuration Management - 
Planning (1) 


Project Class 


EE Configuration Management 
Approach 


Version | Version Control= Documents Documents | Version Control- Documents |x |x |x |x] x 
| __ Version Control- Hardware || x [x [x [X_ 
ptmematccB Tx x 
| Gentine x 
ee 


Configuration Management - 
Planning (2) 


¢ Completes the work started in the proposal 

e Addresses changes from proposal to award 

e Orients the entire team on the process and 
procedures to be used 

¢ Scaled requirements from simple version control 
to full up configuration management 


Configuration Management - 
Execution (1) 


e Maps back through planning and proposal 
development 


e Provides specific Project Class Activities 


e Links directly to the procedures that are 
applicable 


e Informs the project what is assessable and 
auditable 


Configuration Management - 
Execution 


Project Class 
ee eee 
Execution ||| | 
| ProgramManagement || | || 
Sa 
| Maintain risks in Riskacor || |x |x | X_ 


Getgwatonooseres | |_| | | 
Panvesoars x fx | | 
Pan contronaranenene [| [x [x [x 
sraineerng 


Manage Baselines 


Version Control of Work Products Kix tex 


Closing Remarks 


e Process must be organized and tailored to suit the 
needs of the project (within C7Cs business 
processes) 


e No one size fits all 


e Many approaches to process activities —- focus on 
what your company needs 


e Classify based on size and risk 


e Focus on the amount of process rigor that is 
needed for the class level of the project 


Questions? 


Submitted Abstract (1) 


Small to medium organizations, those with staff between 25 and 
250 people <Lessons Learned from Adopting CMMI for Small 
Organizations, Slide 6>, represent unique challenges when trying 
to adopt the tenets of the Capability Maturity Model I ntegration. 
Some of these challenges have been documented in initial studies 
conducted by the SEI and its partners. However, no attention has 
been given to organizations composed of many functional 
disciplines that have a large overall work force but operate 
projects supported by smaller working organizations. These types 
of organizations present unique challenges to adopting many of 
the process areas in a quantitatively and qualitatively managed 
manner. To be successful, C7Cs approach was to recognize that 
different functional disciplines might require multiple procedures to 
complete the tasks, possibly different tools, and definitely different 
terminology. 


Submitted Abstract (2) 


The process areas of Configuration Management, Risk 
Management, and Project Planning are key areas that C7Chas 
found that cross these functional disciplines. These areas provide 
insight into critical differences between sub-organizations. Within 
CM, several challenges exist even with decades of written works 
on the subject. What is important under one discipline or within 
one project may not be in another; how do you define the 
standards to identify what should be managed? Within Project 
Planning, how does an organization identify the roles and 
responsibilities clearly, when they change from contract to 
contract? Do you force one size to fit all or highlight the unique 
needs of each functional discipline? Risk Management is standard, 
but how does a company that has over 280 projects with over 
1400 employees that do not have many similarities organizationally 
analyze them for improvement? The presentation will provide 
recommendations on how to successfully approach these unique 
challenges. 


Submitted Abstract (3) 


e C7Cis a nonprofit, applied research and 
development professional services organization 
with 35 offices. 


CMMI Applications in a Small Setting Finance Organization for 
Process Performance Optimization 


Dr. Mary Anne Herndon 
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Sandra Salars 
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November 15, 2006 
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Sandra Salars 
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Introduction 


Back Ground 


In 2001, the CMMI v.1.1 was released as an integrated model for software development and 
systems engineering applicable to implementing process improvement for both development 
and services organizations. 


In 2002, a leadership team in a small setting, facing multiple short and long term business 
challenges, decided to investigate the feasibility of implementing the practices in the CMMI 


throughout their organization. 


The recommendations of their feasibility study to implement CMMI practices, was endorsed 
by their customer, and the small setting began their process improvement journey in 2002. 


Starting in 2003, the small setting has provided yearly presentations describing a wide range 
of on-going process improvement activities , including planning and implementing of the 
CMMI process areas, and process performance results. 


This presentation concludes this series of presentations and focuses on the benefits of 
implementing CMMI based processes into support functions of finance and contracts and 
subcontractor management. 


2003 Business Case Rationale for Implementing Formal Process Improvement 
Across the Organization 


Business Case Rationale in 2003 Key CMMI Category 
Applicability 


Infrastructure support is expected to provide more | Project Management 
services while minimizing costs over the life cycle 
of technology programs, such as subcontractor 
management, cash flow forecasts and staffing Support 
projections. 


Agility to successfully respond to rapidly changing | Process Management 


business environments and customer priorities 

geet Support 
and requirements is critical to long term 
survivability, including a successful re-compete. 


Process Management 


Planning and controlling life cycle costs needs to | Project Management 
focus on implementing capable/mature processes 
in the support services as well as the technology | Process Management 


programs. Engineering 
Support 


Application of the CMMI model practices and Process Management 


SCAMPIS™ appraisal methods provide a common 
framework to achieve process capability and 
maturity in both the technology programs and 
infrastructure support functions. 


Engineering 


Background of Journey 


Rationale for Initiating Journey 

This small setting faced extreme challenges of 1) maintaining profitability while managing 
increasing performance costs, 2) responding to a dynamically changing customer 
environment and 3) planning for a re-compete of the contract. 


Organization Overview 
This small setting supported their major customer by performing on-site and off-site engineering 
and scientific services and product development for space based applications. 


Kick-Off Activity 

Multi-domain leadership team assembled to plan the multi-year journey to 

higher maturity levels. The initial version of the plan launched pilot projects 

in the small software development organization and concurrent support functions of 
finance, contracts, procurement and HR. 


Initial Obstacles 
The small setting faced initial obstacles of trained resources to implement planned process 
improvements such as processes to integrate key program and technical functions as well as 


Staff training in the CMMI®, performance measurement, and SPC techniques. 
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Progress Improvement Planned Timeline 
2003 - 2006 


Technology Projects 


Process Software & Systems Engineering 
Improvement Class A 
Plan Appraisals 
| ML 2 ML 5 ML 5 


Infrastructure Services 
Contract 


Award 


Quantitative Management = 


Technology —— => Finance 


Transfers 


Leadership 
Team 5 


Finance Organization CMMI Maturity Level 5 Thumbnails 


Process Training Three day Introduction to the CMMI (2004), Statistical 
Sessions Process Control (SPC) and optimization (2004) 


In-house SPC resource provided mentoring (2004 — 
2005) 


Scope of Model Implemented practices from all Maturity Level 5 process 


(CMMI-SE/SW areas 
Version 1.1 Achieved Maturity Level 5 via equivalent staging 
(Continuous) 


Process Model The basis for the process model was the established and 
maintained corporate financial processes. Extensions, 
consistent with the corporate accounting processes, 
included an Excel implementation of revenue optimization 
providing forecasting intervals for: 
ally Monthly headcount and spend rate 
2. Accuracy of an invoicing report 


Measurements Implemented existing required corporate financial 
performance measurements and FTE headcount, spend 
rate and invoicing accuracy forecasting intervals. 


Finance Organization CMMI Maturity Level 5 Thumbnails (Continued) 


Process Resources 1.25 process improvement staff and SPC resource 


One member of the finance organization was the CMMI 
coordinator 


The revenue optimization model developed by the manager 
of the finance organization using individuals and moving 
range. 


Formal appraisals Class C — Spring, 2004, Class B— Summer, 2004 
Class A — March, 2005 


Appraisal Team Systems Engineering, Project Control (Finance), HR, 
Domain Composition | Organization Process Improvement Lead 


(7 team members) 


Example of a Services | The Validation Process Area was implemented using the 
Interpretation of the monthly customer Performance Management Review 
Herel Process (PMR) session to validate the accuracy 
of the receivable report as part of the delivery activity. 
Before the receivable report could function successfully in 
the customer's environment (integrated into the 
customer's financial MIS), the PMR was conducted to 
validate the accuracy and format of this report. 


Body of Evidence 12 affirmation sessions 
500 artifacts/280 affirmations 


Summary of Experiences 


Program control Contracts/subcontracts 
(project performance) (Subcontracting quality and schedule) 


Finance Forecasting Model 


Functioned using a legacy of formal process models for revenue and profit and 
standard performance measurements at the beginning of the CMMI journey 
in 2003. 


Implemented improvements on existing processes, revenue models and 
measurements to expedite journey to develop an integrated process model to 
forecast accuracy of monthly invoice. 


Process performance model included a risk assessment of subcontractor 
performance and project performance. 


Implementation of SPC techniques highly compatible with existing single point 
variance analysis of revenue generation, contractual performance requirements, 
project and subcontractor performance. 


Efficiently transitioned to the use of the CMMI model definitions and continuous 8 
process improvement concepts in improving and implementing forecasting. 


CMMI Implementation Success Factors 


There are a well documented set of process improvement success factors 
seemingly invariant of organizational size, such as higher management 
commitment, resources, appropriate stakeholder involvement in the planning, 
execution of the pilot projects and a compelling business case. 


“HG : Ke) a 


Process 
R Pilot 
Improvement esources 
, Projects 


Five Observations of Process Improvement 
Institutionalization in Small Settings 


In 2002 and now in 2006, the published lessons learned describing process 
improvement success factors focused on larger development and engineering 
organizations. 


From over three years of observations of process improvement institutionalization in 
this small setting: 


A Smaller settings have less formality in their chain of command and their service support 
functions and are usually “one of a kind” closely modeled for a specific customer base. 


2. The resulting flexibility resulting from less organizational formality provides agility to 
rapidly change to meet emerging business challenges. 


2. A less formal environment is also conducive to open communications and a less 
formidable decision making process. 


4. Organizations with open communications typically are more receptive to new ideas and 
have greater motivation to participate in the planning and implementation of new 
processes. 

5. One direct benefit of openness in communications in small settings is the process 


performance models are efficient and the on-going modifications during 
institutionalization periods consume less time. 
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Background 


_ Many organizations have experienced difficulty in maintaining 
CMMI-compliant behavior 

= Projects regress into old habits 

= Senior management turns their attention to other challenges 

= None of the investment pays off 

= When the time comes to re-appraise, you have to start all over 


= This presentation will examine techniques for sustaining and 
capitalizing on your CMMI achievements 


_ ® It focuses on the primary reasons why projects and organizations 
4 fail to institutionalize CMMI practices, and suggest ways to 
overcome them 


NORTHROP GRUMMAN 
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Topics 


#= What institutionalization is — and how to spot it 
= Why institutionalization fails 


= Keys to institutionalization: 
= Addressing the underlying beliefs 
= Accountability 
= Management commitment and support 
= Effective use of audits and appraisals 


NORTHROP GRUMMAN 
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What I nstitutionalization Is 


Institutionalization: The ingrained way of doing business that an 
organization follows routinely as part of its corporate culture. 


When mentioned in the generic 
goal and generic practice 
descriptions, institutionalization 
implies that the process is 
ingrained in the way the work is 
performed and there is 
commitment and consistency to 
performing the process. 


An institutionalized process Is 
more likely to be retained during 
times of stress. 


- CMMI-DEV v1.2 


GG 2 Institutionalize a Managed Process 
GP 2.1 Establish an Organizational Policy 
GP 2.2 Plan the Process 

GP 2.3 Provide Resources 

GP 2.4 Assign Responsibility 

GP 2.5 Train People 

GP 2.6 Manage Configurations 

GP 2.7 Identify and Involve Relevant 


Stakeholders 


GP 2.8 Monitor and Control the Process 
GP 2.9 Objectively Evaluate Adherence 


GP 2.10 Review Status with Higher Level 
Management 


GG 3 Institutionalize a Defined Process 
GP 3.1 Establish a Defined Process 
GP 3.2 Collect Improvement Information 
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Common Features - 
A Lost Perspective in CMMI v1.2! 


Commitment to Perform 
GP 2.1 Establish an Organizational Policy 


Ability to Perform 
GP 2.2 Plan the Process 


GP 2.3 Provide Resources 
GP 2.4 Assign Responsibility 
GP 2.5 Train People 
GF 3.1 Establish a Defined Process 


GP 2.6 Manage Confi 

GP 2.7 Identify and Involve Re 
GP 2.8 Monitor and Cont 

GP 3.2 Collect Improveme 


Verifying Implementation 
GP 2.9 Objectively Evaluate Adherence 
GP 2.10 Review Status with Higher Level Management 
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Organizational Culture 


A pattern of shared basic assumptions that the group learned as it 
solved its problems of external adaptation and internal integration, 
that has worked well enough to be considered valid and, therefore, to 
be taught to new members as the correct way you perceive, think, and 
feel in relation to those problems. 
Artifacts 
= The practices that can be observed in such areas as dress code, 
leadership style, communication processes 
Espoused values 
= The elements the organization says it believes in, the factors that it 
says influence the practices in which it engages 
Basic underlying assumptions 
= Unstated beliefs the organization has come to accept and abide by 


Organizational Culture & Leadership, Edgar H Schein NORTHROP GRUMMAN 
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Standardized Programs 


Standardized programs are programs in which the details of the 
~~ 7 program's processes are always intended to be performed in a specific 

“| sway. They are instituted where it is necessary (or desirable) that the 
results of the program be predictable. They are established for the 
express purpose of the program's means controlling the program's ends. 


-“The Cultural Dimensions Of Standardized Programs”, D. Larry Fraser 


Standardized programs are useful in: 
= Product Consistency 
=" Oversight Facilitation 
=" Expectation Dependability 


NORTHROP GRUMMAN 
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Why I nstitutionalization Fails 


= Few engineers or managers are trained in organizational 
psychology 


= Improvement efforts implement the generic practices (i.e., change 
the artifacts) without understanding or addressing lower level 
contributors to culture 


Covert level 


; Ethics Values Norms 
Intermediate level 


Attitudes Beliefs Priorities 


Overt level Opinions Behavior Conduct Do & Don'ts 


NORTHROP GRUMMAN 
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Addressing the Underlying Beliefs 


= Sponsors and performers must have a strong vision of the 


a 
—" desired culture 
r4 = What are my roles and responsibilities? 
a ty = What changes in behavior are required? 
a = What are the underlying beliefs and values? 


= How do! benefit — WIIFM? 


Covert level 


; Ethics Values Norms 
Intermediate level 


Attitudes Beliefs Priorities 


Overt level Opinions Behavior Conduct Do & Don'ts 


NORTHROP GRUMMAN 
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Five Dimensions of Work 
Reference: Richard Hackman & Greg Oldham, Work Redesign 


# Skill variety - The degree to which the 
work requires you to exercise a 
variety of skills 


= Task identity - The degree to which 
the work requires you to complete a 
whole, identifiable piece of work 


" Task significance - The degree to 
which your work affects others and 
contributes to social welfare 


= Autonomy - The degree to which you 
have control over the means and 
methods you use to perform your 
work 


= Job feedback - The degree to which 
carrying out the work itself provides 
you with direct and clear information 
about how effective you are. 


NORTHROP GRUMMAN 
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Perceptions of the CMMI Common Features 
Based on Work Environment Preferences 


Skill Task Task Autonomy Job 
Variety Identity Significance Feedback 


Commitment to Perform 
= Establish an Org. Policy 


_ Ability to Perform 

= Plan the Process 

= Provide Resources 

= Assign Responsibility 

= Train People 

= Establish a Defined Proces 


Directing Implementation 
= Manage Configurations 
= Involve Stakeholders 
= Monitor/Control the Proces 
= Collect Improvement Info 


_ Verification 
= Obj. Evaluate Adherence 
= Review with Higher Mgmt 


“Aligning CMMI Strategies with Individual, Project, and Organizational NORTHROP GRUMMAN 
11 Perspectives,” Software Technology Conference, 2003 
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Key Messages OBS — _ 
XU 


*4 = Achieving and maintaining mature processes LA 
— is essential to meeting our business goals | V 
Fi = Predictability, performance, quality yy 


. = CMMI involves short-term investment for long-term gain 


=" CMMI is an enabler (not a guarantee) of project success 


= Other aspects (people, technology, customer relationship, etc.) are 
equally important 
= The value is often risk reduction (which may be difficult to measure) 


= CMMI is aa set of proven, industry best-practices 
= Adoption is about learning how to apply these practices to our work 


= The practices may feel awkward and have limited value until we 
learn them 


= |t’s OK to make mistakes — we will get better over time 
= When the entire organization is behaving maturely, everyone’s job 
becomes easier 
=" Continuous improvement is a way of life 
NORTHROP GRUMMAN 
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Accountability 


*Z = Adopting and sustaining CMMI is about each practitioner learning 
<7 and performing the new behaviors 


_ 4 © The role of management in cultural change is to hold people 
, | accountable for the new behaviors and conduct 


= Change agents can enable management by: 
= Helping them have a clear vision of the new culture 
= Identifying inappropriate behavior 
= Providing tangible, objective 


measures of adoption/sustainment ow. 


Tray fan ly Adlapy Sry y Late Mey, 
Yn, 


00, 
“sony Wo 


; . "Crossing The Chasm“, Geoffrey Moore 


NORTHROP GRUMMAN 
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Management Commitment and Support 


Management: 
= Understands the key messages 
= Is willing to take actions to reinforce them 


= Provides resources to support/sustain process improvement 
efforts 


= Sets expectations that essential project functions will be funded 
and processes will be followed 
= Project planning, estimation, tailoring, CM, QA, etc. 


= Supports process improvement and sustainment, rather than 
passing appraisals 


=" Rewards mature processes development and sustainment rather 
than individual heroics 


= Tell me how you will reward me, and I'll tell how | will behave 


NORTHROP GRUMMAN 
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Effective Use of Audits and Appraisals 


= Process and product audits provide tangible, objective 

S measures of adoption/sustainment 

= Policies, processes, and standards must reflect the desired 
behaviors 


= Appraisals evaluate the effectiveness of the audit program 
= Standardized tools, approaches, and methods 
= Consistency of appraisers — if they understand the way we are 
structured and operate, there is less time required to understand 
what we are doing. 
= Pre-appraisal activities to prepare projects for the appraisal 
process 


= The frequency of audits and appraisals, and the sampling, must 
reflect the progress of the cultural change 
= As the culture begins the change, more frequent and more in-depth 
audits/appraisals are required 
= Later, the amount of audits/appraisal may decrease, if the culture 
has truly changed 


NORTHROP GRUMMAN 
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Summary 


Process improvement can be sustained by understanding and focusing 
on institutionalization 


f = Addressing the underlying beliefs 
a = Accountability 
, | = Management commitment and support 


: = Effective use of audits and appraisals 


Commitment to Perform 
GP 2.1 Establish an Organizational Policy 


GP 2.2 Plan the Process 
GP 2.3 Provide Resources 
GP 2.4 Assign Responsibility 
GP 2.5 Train People 
G »3.1 Establish a Defined Process 


) Ability to Perform 


GP 2.7 Identify and Involve Relev 
GP 2.8 Monitor and Control t 
GP 3.2 Collect Improvement In 


Verifying Implementation 
GP 2.9 Objectively Evaluate Adherence 
GP 2.10 Review Status with Higher Level Management 


NORTHROP GRUMMAN 
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Maintaining Institutionalization After the Appraisal Team Leaves 
BAE Systems 
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Nashua, New Hampshire 
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Presentation Agenda 


—Introduction 
—The Organizational Facts 
—Experience Basis for Premise and Presented Approach 


— “Elastic Band Phenomenon” 
—Comfort Zone Scenarios 


—Making It Easy To Do The Right Things 
—Conclusion 


—Final Thoughts 


Electronics & Integrated Solutions Operating Group 


Electronic Warfare BAE SYSTEMS 
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Merrimack 
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Yonkers 
Fort Worth 


Sensor Systems BAE SYSTEMS 


Manassas Jerusalem 
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Merrimack Nashua Lexington Washington, D.C. 


Experience Basis for Premise and Presented Approach 


—Twelve years experience with appraisals and 
assessments 
—Four Software CMM® assessments 
—Two EIA / 1S 731 appraisals 
—Two CMMI® appraisals 


—SEI certified Introduction to CMMI® instructor on 
Enterprise Process Group staff 


—Enterprise Process Group participant on SEI SCAMPIS™ 
upgrade team 


Process Development Time Line BAE SYSTEMS 


Continue evolving process based on process improvement suggestions 


2006 
Further expanding use of process 


Expanded Level 4 and 5 behaviors from systems and software 
2005 Implemented process improvement to enhance process usability 


Class A appraisal assesses process (CMMI® SE | SW) at Level 3 


Class B appraisal assesses process (cMmi® SE /SW/IPPD/ SS) at Level 4 
Integrated Product Development Process implemented | 


2004 


Enterprise Process Group officially formed 
Management and technical (SE / SW) processes developed 


Customer requirements for 


2003 


CMMI® appearing in RFPs 


Systems Engineering Process assessed at EIA / IS - 731 Level 3 
2002 SWprocess updated to SW-CMM Level 5 
Development of an integrated process begins 


Gap analysis performed for engineering and program management processes 


2001 

Software process assessed at SW-CMM Level 4 
onee BAE SYSTEMS Lifecycle Management 
1998 Software process assessed at SW-CMM Level 3 eae keel: ; 
1994 CMMI® released by SEI 
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The “Elastic Band Phenomenon” 


—The Elastic Band Phenomenon Premise 


—Once a formal appraisal or audit has been 
completed, the high energy and extra 
attention to process compliance will naturally 
wane, and the organization will tend to 
migrate toward the pre-appraisal comfort 
zone 


Comfort Zone Scenarios BAE SYSTEMS 


The appraisal coverage (AC) is less than the : 
organization’s comfort zone (CZ) Low risk 


— Following the appraisal, the organization remains in 
its comfort zone 


The appraisal coverage is equal to the organization’s 
comfort zone 


— Following the appraisal, the organization remains in 
its comfort zone 


The appraisal coverage stretches the organization’s 
comfort zone 
— Following the appraisal, the organization reverts Stretch goal 
back to some point less than its original comfort eh ss | 
zone (Elastic Band Phenomenon) CZ 


Post-Appraisal Organizational Impact 
(Reduced Comfort Zone) 


Any reduction to process compliance emphasis may result in: 
¢ Non-uniform application of the process 
- Reduced familiarity with the process through non-use 
- Increased cost of rework for process compliance 
¢ Confusion as to when it is right to follow the process 
- Impression that compliance is only needed to pass appraisals 


e Practitioners’ frustration with management for providing mixed signals on 
process compliance 
- Partial/Minimal process compliance; quality suspect 


Potential for Poor Project Planning 
and Poor Project Execution 


Making It Easy To Do The Right Things 


—To facilitate institutionalization, the Enterprise Process Group 
ensures the process is easy to use 


—Approaches/techniques we are using: 
— Process marketing 
— Ease of access 
— Process training 
— Facilitated project start-up 
— Planning work product reviews 
— Process automation and process aids 
— Internal process assessments 
— Active process ombudsman 
— Web-based process tips 
— Solicited process improvement ideas 


Process Marketing BAE SYSTEMS 


— Display eye-catching posters SS ae 
around your company BRUNE YX 


— Instill a message: 
“Bring Your Team Together” 


— Reinforce with a slogan: 


“Unifying Teams Through 
Integrated Processes” 


— Develop a logo: 
“IPDP” is our Integrated Product 
Development Process 


Unitying Teams through integrated Processes\._ 


Ease of Access BAE SYSTEMS 


Address 2 http://www.usa02.na.baesystems.comipdpjindex.htm a Go Link 
Integrated Product Development Process SAESYSTEMS 
Unifying Teams Through Integrated Processes 


IPDP Home 
EalS Today 


Engineering Process for 
Small Projects (EPSP) 


Process Assets Library 


Welcome to your IPDP website! 


Check out the Streamlined Process, updated Activity Flows, and Process Tips! 


Summary of Recent Changes to IPDP: Ver 2.3 #0» 


= Release Features 


About IPDP 

IPDP Process = Update to Release Notes 

IPDP Help = Release Notes 

Ask the EPG = Summary of Changes “IPDP...start programs Green, keep them Green” 
IPDP Evolution: se 
Year in Review: Click here to see the results of the latest improvements. Evolution 


_Strearalintng ihe wherprise process 
Note of the Day: 
Measurement Plan: Describes the details of 


the content, frequency and method of analysis 
of measurement data. 


Institute a comprehensive Web site that is easily navigated 
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Process Training BAE SYSTEMS 


—Effective training is key to the success of a process 
—Multiple levels of training required 


—Process introduction course 
—General material and recent changes 
—Annual refresher required 


—Integrated training for both the planning and execution phases 
—Includes project management and technical management 


—Related training for Peer Reviews, Decision Making, Risk Management, 
and Supplier Management 


Facilitated Project Start-up 


—The Enterprise Process Group and the Project 
Team coordinate the launch activities to develop a 
comprehensive project plan in a timely manner 


—The Project Launch mentoring environment 
—Is a series of modules designed around project 
planning 
—Provides reinforcement of the process 
—Scopes the process based on project attributes 


—Facilitates planning work product development 
with project team leads 


Planning Work Product Reviews 


—The Enterprise Process Group Is actively engaged in 
the project’s planning activities 


-Guides planning work product development during Project 
Launch 


-Participates as a stakeholder 


-Checks planning work products compliance 


- Review lists guide the development of work products and are 
used as the checklists for audits 


Enterprise Process Group Partners with the 
Project Team during Project Planning 


Process Automation and Process Aids BAE SYSTEMS 


P r Oo ces S C U S to m ] Z a ti O n A | d PSP INFORMATION | PROJECT ATTRIBLITES | PROJECT ATTRIBUTES (CONT) | TEMPLATE INFORMATION | 
EXIT 


_ U SES a SE rl eS of q uestl O NS to h el p ~ ELEMENTS ~ PHASE REVIEWS — SYSTEMS ANALYSIS AND DESIGN — 
1 . : : Systems for this project will have: 
Select the system Which phase reviews will 
the project team select the seiuectearieey ||| Valance (check all hat 991) 
1 to this project (check all that apply) Analysis: Design: 
applicable process elements (eneck al that apy) 
7 Phase Review - M Structured M Structured 
Me oyster 7 Phase Review - Object-Oriented [ Object-Oriented 
. . ae SO 7 Phase Review « Adapted MV Adapted 
— Establishes the standard project oh aaeieeac ass 
‘ Phase Review - SW ANALYSIS AND DESIGN - 
shared area Software F Bites Revie Software fr tis projec wil have 
: (check all that apply) 
iV Phase Rewiew - Analysis: Design: 
Mechanical iV Phase Review - IV Structured IV Structured 
—Pre-populates project information Pe Suepomabiby | Phate Renew C Pare oneae We Sieae ies 
. . . o x as Adapted Adapted 
within the planning work products - SUBCONTRACTORS | 
Will this project have any major development IV&V 


subcontractors (>$3M)? Will this project have a deliverable IV&V plan? 


@ Yes © No @ Yes No 


NEXT >> 


Automation facilitates process use and compliance 


Process Automation and Process Aids BAE SYSTEMS 


Project SharePoint Template 


— Enables a collaborative team 
environment 


— Supports and automates aspects 


of the process a ————— 
-Issues & actions tracking sree) | 5 o~ me 


a 


-Project calendar 


— Provides a single point of access 
for project information 


— Common format allows new 
project personnel to quickly find 
project information 


Process Automation and Process Aids 


BAE SYSTEMS 


Responsibility Assignment 
Matrix (RAM) Aid 
—Facilitates preparation for 
project meetings and work 
product reviews 


— Ensures accurate stakeholder 
involvement for specific reviews 


— Automates meeting invitations 
using MS Outlook 


— Creates agenda, attendee list 
and minutes template 


|= 


: 
| 


Legend for Meetings and Work Product 
Reviews 

0 = Roles Developing the VVork Product 

R = Roles Expected to Review/Attend 

4, = Roles Required to Review & Approve 

x = Review Meeting Participant 


4 
Project Role \_Istakeholder Name 
Project IMT & Functional Leads = S 
Project Manager 

AuUcioe@@Wianagement Lead 
Information Management Lead 


ks | om 
Ai 


Project Engineering Manager (PEM 
Operations Program Manager (OPM 
MS&PA4, Lead 


NININ|=la/alaia 
NO) ls aan 


System Engineering Manager (SEM 
Software Engineering Manager 
Electronic Engineering Manager 
Mechanical Engineering Manager 


Supportability Engineering Manager 
Risk Officer 
sother= 


NIN NN 
)5 R88 
A 


integrated product team name A> 
IPT Lead 

Sysiness Management Lead 

Informa ement Lead 


Nis 
o/c 
D 
5 
iy 
2 


30 
I4 


View Specific Information + 


Ready 


Subcontract Admi o 


MS&PA Representative /Qual™Gmgineer 
i ii ines hAorns, CDEhAY —_ i —— 
4 > > ‘ Cover }.RAM / jadi 


Check Stakeholder Email(s) 


in 


Set Up Meeting 


Analyze RAM 


Internal Process Assessments BAE SYSTEMS 


—Align with CMMI® and are independently conducted 
-Performed on individual projects 


- Staffed with people knowledgeable in the CMMI® and the organizational 
process 


-May be conducted periodically on projects with long durations 
- Similar to a Class B (without the organizational elements) 


—Check for compliance with CMMI® and the organizational process 
—Assess application and understanding of the process 


—Indicate the effectiveness of: 
-Process institutionalization 
-Process training 
-Project start-up process 


Active Process Ombudsman BAE SYSTEMS 


—Web-based mechanism to ask process-related questions 
—How does the process apply? 
—Where do | find...? 
—Can you tell me how this practice relates to my project? 


— Quick response Is provided 
—Process Ombudsman confirms content and intent with requestor 
—Answer is developed and vetted with process team 
—Response sent back to requestor 


—Web-based Frequently Asked Questions (FAQs) developed from 
Process Ombudsman activity 


Web-based Process Tips 


—Process Tips provide quick access 


Making Decisions involves a formal evaluation process to 


o Bee (OND 

ti Fi 5 || identity, evaluate, and select alternative solutions for critical 

to a itl O Nn a g Uu | an ce : iy decisions that have multiple alternatives and multiple critena 
4 1 A formal evaluation process reduces the subjective nature of the 
F d ifi t 1 decision and has a higher probability of selecting a solution that 

—_ OC U se O Nn S pe Cl IC p rocess O p | CS a meets the multiple demands of relevant stakeholders. 
A formal evaluation process can vary in formality, type of eriteria, 
and methods. Less formal decisions can be analyzed in a few 


hours and result in a short report. More formal decisions may 
fequire separate plans, months of effort, meetings, sirnulations, 


_ Pp O rtal to th e p roce SS ; prototypes, piloting, and extensive documentation 


Multiple methods exist for decision making Each method has Make Decisions Practice 
Situations for which il is better suited, For more information 


— Provides interpretation of process ee 


Perfor Stakeholder 


inte nt , y ? : =i Should | Use the Make Decisions Practice? oviow List 
or Perform Approval Review 
4 Tharitl ; a decision that can cause schedule delays over a certain 
— Reinforces process definitions Sere percentage o1specfe amount atime 


Assets Library (PAL) for a decision that can drive a favorable cost to benefit ratio 


— Re | nforces corre ct behavior paee of tade between this process and the decision's impact 


2 decision directly related to topics considered medium or 
— high risk 


a Al e rts of CO mM mM O Nn a decision that selects a supplier 
m iSU n de rstan d r n gs a decision that selects COTS or reuse products 


a decision that can resolve requirements conflicts 


. k | d . a decision about system architecture selection 
—_ Li nN S to re ate ite mS a decision that selects among allemative engineering 


support environment choices 
@ make-buy decision 


a decision related to changing work products under forrnal 
Configuration control 


Your project must make the choices. 
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Solicited Process Improvement Ideas BAE SYSTEMS 


—Enterprise Process Group actively solicits improvement suggestions from 
users 
— Accomplished directly through the Process Web site 
— Rewards for best suggestions 
— Results in process updates 


—User involvement facilitates buy-in 
— Enables best practices 
— Encourages continuing inputs 
— Improved process enhances process institutionalization 


Institutionalization is Enabled through 
User Involvement 


Institutionalization....Raising the Bar 
(Expanding the Comfort Zone) 


—Expand the use of process to a wider spectrum of projects 


— Familiarize the largest possible audience with the process and the process 
terminology 


—Leverage process, by its use, during proposal preparation 


— Ease transition between the proposal, and project planning and execution 


-Development of planning artifacts, technical plans, top-level system 
architecture that are aligned or compliant with the process 


— Design quality in (by use of the process) from the beginning 


Reduces future Elastic Band Phenomena 


Conclusion BAE SYSTEMS 


—Establish an organizational standard process 
—It must be comprehensive, concise, coherent; yet easily usable 


—Be watchful of the organization being pulled out of its comfort zone, and 
then relaxing to a previous level or lower (Elastic Band Phenomenon) 


—Provide process support and process aids to reduce the tendency to 
revert to a relaxed state 


—To make the process easier to understand and use correctly 
—To make compliance routine; not unique (institutionalization) 


—Implement process improvement as part of the process 
—Methods to suggest improvements must be easy to use 
— Suggestions must be embraced as improvement opportunities 


Final Thoughts 


BAE SYSTEMS 


—Questions? 


—Comments? 


Contact Information BAE SYSTEMS 
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Enterprise Process Group 
Performance Excellence 


BAE Systems 

Electronics and Integrated Solutions 
Nashua, New Hampshire 03061-0868 
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Tel: 1 (603) 885-7274 
email:john.m.johnston@baesystems.com 
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Tomorrow’s 
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Missile Systems 


Does this look Familiar??? 


Process for the Sake of Process Helps No One! 
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Missile Systems 


Users Perspective on Process 


=» TOO many conflicting processes 


» No consistency among directives and 
processes 


a My functional organization tells me one 
thing, but my program another 


= Waste of time and money 

=» Process gets in the way of “real work” 

s What does this process stuff really do 
anyway? . 

= | am behind schedule and don't have time 
for process stuff 


With So Much Process, When Can | Get My Work Done? 
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Program Manager’s Perspective on 7 ahee 
Process issile Systems 


=» TOO Expensive 

=» TOO much process, not enough budget! 

= | don’t need to use process, | make this 
company money without it! 

s What does this process stuff really do 
anyway? 

=» |!am meeting schedule and budget, 
process just gets in the way! 

a Why does the organization tell me to use 
process without paying for it? 

=» My Customer is not going to pay for 
process work! 


| Am Meeting Schedule and Budget, Process Just Gets in the Way! 


11/28/2006 | Page 5 


Missile Systems 


EPG’s Perspective on Process 


» Conflicting initiatives 

s All things for all people 

» Continuous new direction 

= Flavor-of-the-day response 
» Passive sponsorship 


Continual “Redirection” Leads to Confusion and Ineffectiveness 
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Our Moment of Courage: What Did Not Work ““*"°°*"s 

=» Our Customer created a new 
set of requirements for 
certain types of programs. 


» Our Company decided to 
create Corporate-Level 
Initiatives to answer our 
Customer’s needs. 


» Our Organization created a 
whole new set of directives 
to answer the need, without 
leveraging all the efforts that 
had already been created as 
part of AS90001, CMMI...etc. 


A Process Created in Isolation is a Waste of Time 
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Raytheon 
_WhatDidNotWork 


= Silo Processes 

= No Big Picture 

» New initiative = new process 
» Poor Communication 

=» No Accountability 

» Unorganized Deployment 

» No common agenda, too many hidden agendas 
» Assumed compliance 


Creating New Process for Every New Initiative Costs Too Much 
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Our Success Stories dia 
=» The new directive system looks beyond 
initial charter 
a Synergy by using an Enterprise-wide 
Change Review Board (CRB) 
=» The organizational horsepower to make decisions! 


= All new initiatives now have a place 
where all interested parties can come 
together and knowledge share/ 
collaborate. 


When We Leverage Off What Already Exists, We Do It Better 
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Raytheon 


» No new initiative without CRB 
reviewing It 
» Enterprise wide vision and voice 


» Leveraging of existing process 
and infrastructure 


a Strong communication plan 


» Accountability at all levels to ensure 
successful deployment 


Knowledge Sharing Across the Enterprise 
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Customer Success Is Our Mission 


Questions?? 


Thank you for your participation 


Presentation Outline 


< 


Vi. 


Introduction 
Scene/Act 

a) User Perspective 

b) Program Perspective 
What did not work 

a) Courage/RDS system 
What did work/success 

a Visibility into deployment 

b) Programs using PD, Tailoring, Gates 

c) RDS work in Progress 
Lessons Learned 
Questions 


Missile Systems 
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Agenda 


e Exponential Growth in Software 
— Expansion in Software Process Assets 
¢ Current State of Practice 


— Large Organizational Standard 
Processes (OSPs) 


— Manual Tailoring of OSPs to Project 
Define Processes 


¢ Problems with Manual Tailoring 


e Automated Rules-Based Tailoring 


e Lessons Learn, Contributions & “Perfect Storm” Event, October 1991 
National Oceanic & Atmospheric Administration 
Road Ahead 
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Exponential Growth in Software Usage & Assets 


* Provided by Lockheed Martin 
In The Beginning 


m Ah i iH 


pce ee Sioa 


1980’s 


1970’s 


F-15A 
50,000 
LOC 


Market Dynamics: Drivers That Increase the 
Demand on Additional Systems & Software Assets 


EEE I 
Platform eer ree Enterprise 


Acquisition Model 


Dominant Strategic 
Prime Program Execution Teaming 


“Boxes” Int tion Chall — . 
ntegration Challenge Stacks” 


Proprietay ——————— Plug & Play 


Architectures and Standards 
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Example: Shift to Service Delivery versus 
Produce and Delivery Perspective 


vies 
Services represent 80% of the US economy (Source: 
Paulson, Linda. “Services Science: A New Field for Today’s 
Economy.” IEEE Computer Society, August 2006). 
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Expansion in Process Assets Versus UsSage* 


What size problem can a given number of people attack, 
using different sizes of organizational assets? 


Many people (using a heavier set of assets) 


few people 
rN 


Many people 
(using a very heavy 
set of assets) 


Many péople (using a light set of assets) 
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Size of Organizational Assets 
*Slide adapted from Alistair Cockburn, Presentation at SSCI, 3/29/06 


Tailoring the Corporate OSPs to Lighter Project 
Defined Processes 


Organization’s 
Measurement 
Repository 


OPD Organization’s Set Life-Cycle Model 
of Standard Processes Descriptions 
Process 
Architectures — 
Organizational Assets 
Tailoring 
Guidelines 


LIEGE 

Organization’s 
Process 

Asset Librar 


Project Environment 


Project A’s Project B’s 
Defined Process Defined Process 


~ Project C’s 
Defined Process 
IPM IPM 
Project A’s Project B’s Project C’s 
Project Plan Project Plan Project Plan 


*Training material registered in the U.S. Patent & Trademark Office by 
Carnegie Mellon University. 


Tailoring Example: Software & Systems Engineering 
Processes & Practices 


Manual ° ; 
7 orporate 
Gov’t Std Tailoring Standards & Other 
Oviiet Command Media 
CMMI Domain Specific Standards | | |f | Sa 
V1.1 & V1.2 
Organizational 
Standard 
ISO/IEC-12207 Process process(es) 
Standards 
Manual 
Tailoring 


Project 
Defined 
Process 


Project Specific Standards 


Business Units 
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In Theory, These Steps Should Work: 


1. Classify program = 7. Assemble program 
using characteristics D defined process 


4. Find sources 
2. Select analogous for uncovered 
programs based processes (OSPs 
with similar or other) 
characteristics 


6. Tailor processes 
to unique 
program needs 


5. Select sources 


3. Identify for process tailoring 


processes with no 
coverage 


The difference in “Theory” and “Practice” is that in theory 
they are the same however in practice they often are not 
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Acknowledged Problems with Manually Tailoring 


¢ Low perceive value by program managers 


e Insufficient knowledge and implementation of tailoring 
rules 


e Too much data 


— Organization Standard Process consists of hundreds of process 
elements; thus, difficult to determine which ones are applicable 
to their project 


e Insufficient planning time available 


e Lack of qualified resources 


— Organizations who are lucky enough to have professional 
members of their technical staff who can manually perform the 
needed tailoring are often not available 
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Potential Solution: Automated Rules-Based 
Tailoring Engine 


What Is the feasibility to using an intelligent agent (e.g. 
Tailoring Engine) to inform and direct the selection and 
tailoring of the OSP based on a set of business rules? 


a A 


Y 
>» J 
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Proof of Concept Approach 


e Research the field (please see references) 
— Identified key issues and risks A itn 
e Selected a “Proof of Concept” approach Se | 
e Develop an engagement plan 
— Defined the problem statement among key stake holders 


e Defined the type of OSP we wanted to tailor based on automated 
process tailoring needs 


e Selected a publicly available, accepted and reasonably complete 
functional process guide as a mock OSP 


¢ Defined the type of tailoring tool we needed based on our tailoring 
requirements 


e Selected the tailoring tool 


¢ Developed a set of tailoring rules for the proof of concept 
demonstration 


« Collected contributions and lessons learned 
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The Formulation of Effective Tailoring Rules 
Challenging Due to Project Asset Attributes 


Common Source e Project Characteristics (cont’d) 
Standards — How complex is the system? 
— (#COTS, #Shalls, #Ilnterfaces, 
« Project Characteristics #Users, SOS, #HWCls, #SWCIs, 
— Are requirements well DP rqts) 
known? (Yes: Waterfall, — System Composition (HW, SW, 
O&M or Incremental life both) 
cycles; No: Prototyping, — Product Intent (Feasibility Study, 
Incremental, or R&D, Operational Program) 
Evolutionary life cycles) — Contract Type (FFP, T&M, 


CPFF, Award Fee) 


— Scope (Formality, Control, 
Structure) 


— Security Requirements? 
— System Size (S, M, L) 
— Level of Risk 


OSP Example: Systems Engineering Standards 
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Interdependence of Engineering Sub-processes: 
Naval Systems Engineering Guide and EIA 632 


Sub-process 14 Sub-process 15 


Acquirer 
Requirements 
Sub-process 16 


System Technical Requirements 


ASSIGNED TO 


Logical Solution Representations 
Sub-process 18 
Derived Technical Requirements 
Physical Solution Representations 


SOURCE OF 


Sub-process 19 ; ; 
Design Solution 
Specified Requirements 


Requirements 
D TO* Definition 
Process 


Solutions 
DTO Definition 
Process 


Sub-process 17 


*Those requirements not 
assigned to Logical Solution 
Representations 
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SOFTWARE 
CONSORTIUM 


13 


BURLDING 
SOLUTIONS TOGETHER 


Deceptive Similarity of Industry Standards 


¢e Because the source standards cover roughly the same 
ground, we anticipated that “normalizing” them would be 
relatively straightforward 
¢ We expected to find (and did!) differences in: 
— Scope, terminology and level of detail 
¢ Subtler differences emerged in the source standards’ 
codification of: 
— Product life cycle 
— Development life cycle model 
— Recursive application of process throughout a system hierarchy 
e Industry standards do not cover all the necessary 
processes (e.g. finance, legal, ethics, etc) 
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“Inter-connectiveness” of Process Steps 


Acquirer 
Requirements 
(SP14) 
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SP 22 Systems 
Analysis 
Process, 

SP26 Requirements 
Validation 
Process 


SP 2 Acquisition 
Process, 

SP4/5/7 Planning 
Process, 

SP 10/11 Assessment 
Process, 

SP 12 Control Process, 

SP 16 Requirements 


Definition Process, 


SP22 Systems Analysis 
Process, 
SP 26 Requirements 


Validation Process, 


SP 31 System 
Verification 
Process, 

SP 33 End Products 
Validation Process 


ICD, 

CDD/ORD, 

Engineering 
Investigation 
Reports, 

Utilization & 
Readiness 
Reports, 

Specifications from 
higher level 
system building 
blocks, 

Sponsor high level 
operational 
concept graphic 
architecture 
(EXT), 

Effectiveness analysis 
reports (SP 22), 

Effectiveness models 
(SP 22), 

Acquirers 
Requirements 
Validation 
Revisions (SP 26) 
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ICD(SP 
2/4/7/10/11/16/3 
1/33, 

Effectiveness 
Analysis 
Request (SP 
22), 

Measurement of 
Effectiveness 

(SP 5/7/16), 

CDD or CPD(SP 
2/4/7/10/11/16/3 
1/33), 

Specifications from 
higher level 
system 
building blocks 
(SP 16), 

Acquirers 

Requirements( 

SP 5/16/26) 


Decomposition - Small Changes in a System 
Engineering Standard Drive Many Other Changes 


Life Cycle Phase Definition 


Product Design 


Project 
Inputs Entry Activities Exit Outputs 


Pp | an Criteria Criteria 


Requirements Requirement Determine Technical Solution De he Design documentation 


Using the DAR procedure (TSS_CMMI_-PRC-DAR), evaluate and select the Eee STU E 


epee a technical solutions in accordance with the PMP. 
j seems Retuicrpes tease Using the DAR Procedure, decide whether to make, buy, ore use Diced Newent 
Requirements documentation updated Race neh ‘ y Specifications, 


‘operational concepts 


3, Update the project plan and WBS, if appropriate Geen 


High-level Design 


1 

and interface requirements) into components and update the RTM, 
2, Collect and an alyze existing system/product documentation. 
3 
4 


Develop and maintain the high-level design including interfaces, 
Peer Review the high -level design using the Peer Review Procedure 
(TSS_CMMI -PRC-PEER_REVIEW). 

S: Review the high -level design with stakeholders, as needed 


Develop 


Integrate 
1 
2 IAT i rage aed eee ese aera eee 
RIM 
3 Tete ae ates oe canal es ot nae 
A Define mel cecntie HIN eel ovuster of anebonrn eal To 
oe 
5 (Le cores ies aveac meme eee 
solsie heen acne 
6 Deseealgstinn ead er trig. 
5 ee dren emaeainue eee 
ee 
8 
Closeout 


Detailed Design 


Decompose high -level desiga into the lowest level product components. 


Documentation 


Complete design documentation. 5 
9, Peer review design components using the Peer Review Procedure. 

10, Execute the Baseline Procedure (TSS_CMMI _ -PRC-BASELINE), 

11 Review the final design with relevant stakeholders, 

12, Update development folders. 


B. Update requirements, operational concepts, and scenarios with information 
learned during design 


Change Control 


Management Oversight 
Tracking 


Specification 


Peer ReviewProvedure 


Purpose 
‘as rocdre ove nein of emdacng prac fre prod 


Responsibiliy/tuathoriy 
tier 


Project Review Comments'Summary Ferm ‘Special Considerations 
The ys afgerreriens ax dtd bik vay in may. Thtype of gearrerien Yous shouldbe 
wenuntu Taed hei xd complet ofthe ea prod beng revered aio to otar leet cs. AL 
robert Information, ‘ypescfpernerien ne seta condoted wih ceay dertood ole, osama ad garde 
Feeds Nome Vlieigueeies Wives naa au nam we ann swpatel messuraant data, The saps ofthe procedre define these arictte & « vel tha oplies toallpeer 
| ‘pei, Sper dedguer qpmpiits te te tine pee efperveders we dined m«ublfelowite te 


Vigo of Predert Ses fina toy wrevel Tociomer tin opmanel rods maw Wteo onl ators tor mds wa denge Fe pm oe roe supe 


‘a ~~ Template 


Wed redut Completion ofc prot 
Review Information # agpiobh tans saponing wok pode, # dreicessrvre produ & ible thes 
mar ‘entwined i peels) 


Outputs Bait Criteria 
+ Comphted Peer Reviow + Bate work product within scope ofthe reer | 
CommemsSaamy Fem ashen reered 
Total Indiv bual Esasminatin time ms wm wy md + Redinedarkopis if lick) + Mess dt, Reser ConmoeiSeay Fa, 


Life Cycle, 
Processes and 
tandards 


Parsicipant Information sn 
Xm 


Acivites 


Prepare for Peer Review 


p} After detenining that he ery ctais hevebeensn, prepareresiey page. 


t [NOTE: Hike reviewed wou produ «doom, gelled. Hai canparion ams edd to 
| i ul povidesuiing domme evra, chide Revi Crome Simms Farm Freer: 


Sep 2 enify quite reiew putidpanis nd wsign rales 


Sep 2 Gfodeatr) Arrangerevewlegisks,Neity reir purtcipans of date, ine, nd Wein tthe 
ony typeof pea rete, pariipanb role, aiew cea reiay scope, and wny prow 
infernaton. 
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ae 


Tailoring Effort Increases Non-Linearly with 
Program and Technology Complexity 


Program/Project 
Management 
Scope 


3. Array 
Program or 
Set of projects 


2. System 
Project with 
complex set of 
interactive 
elements 


1.Assembly 
Project 
consisting of 
single unit 


Increasing: 
Size 

Scope Control 
Planning 
Subcontracting 
Documentation 
Bureaucracy 


Tailoring effort 
increases non-linearly | » 
a, 
- 


Ze 


Increasing: 
Multi-Systems Planning 
Systems Engineering 
stems Integration 
qPnigurater Management 
esign Cycles 
| Analysis & Management 


| 


7 \ncreasing: 


Technical Skills 

Flexibility 

Developmet & Testing 
Late Design Freeze 
Technical Communication 
Risk & Opportunity 


Source: Shenhar and Wideman Established Mostly Advanced Highly Advanced 
Established Or Exploratory 
(Classic Tech) (Medium Tech) (Hi-Tech) (Super Hi-Tech) 


Technology Content 
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What We Determined: 
Devil in the Details 


e State of Practice 


— Some on-going research in the area of automated 
rules-based test engines 


— Existence of a limited set of reasonably mature 
automated tailoring tools 
¢ Significant manual-tuning required dealing with most problems 
e Limited number of “universal” tailoring rules 
¢ Output often compatible with Microsoft tools 
— WBS and Project Plans automatically generated 
— OSPs nominally contain a substantial body of 
knowledge and data 
e Many different formats and types of artifacts 
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What We Determined: es 
Devil in the Details Ys 


¢ Technically Pe com 
— In general, standard processes are not spaineacdecieae: 
for tailoring 
e Makes automation difficult 


— Organizational assets are expanding, costly to maintain, and are 
becoming more difficult to tailor to projects 


¢ Difficult for an OSP to cover every case and be lean 
— Tailoring rules difficult to establish 
e Numerous attributes drive the process tailoring decisions 
e Strong demand for the establishment of robust tailoring rules 
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What We Determined: 
Devil in the Details 


¢ Technically (Con’t) 


— Multiple frameworks (e.g. CMMI, ISO, ITIL, etc) are not paces 
or easily customizable 


¢ Different frameworks need different data elements 
¢ Organizationally 
— Corporate committed, projects involved 
¢ Human element 
— Natural tendency to tailor the OSP based on least resistance 
e Information from similar programs 
e Reliance on local process improvement teams 
— Difficult to demonstrate to program managers the ROI 
— Lack of consistency in process definition for most process areas 
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Road Ahead 


¢ The problem is real as 
— The problem is not going away and will grow in nmonances with 
time 
e Inthe short run, tailoring responsibility will nominally rest 
with the process engineering teams 


— Key focus will be the development of tailoring rules and 
guidelines 


— Rules-based engine research will continue 


e Inthe longer term: 


— Organizations will use process architectures to design and 
populate their process asset libraries 


— Intelligent agents/rules-based test engines will become more 
prevalent 
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Road Ahead 


¢ Utopia Perspective 


— What could we do differently if we started over? 


e Secret of success may be a shift of focus from assets to 
process execution information 


e Approach 


Define a common set of software and system engineering 
processes for the organization 


Map the common set of processes to the organizations lifecycles 


Apply processes/lifecycles on programs and collect data 


Store process execution information in the PAL 


Improve processes based on results and establish tailoring rules 
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Road Ahead: Example - Service-Oriented 
Architecture & Asset Management 


- Service-Oriented Architecture is an approach to building IT 
systems out of common parts 
— Represents a breakthrough in the way we build IT systems 
¢ Composed of reusable components, called services 
e Service is a building block that performs a distinct function 
— Evolution of client/server architecture 


¢ Functions of user I/F, application logic and data management 
are separated and decomposed still further 


Why Now 
— Internet and World Wide Web ————_> 
— Business/Quality Focused —————>|___ 
— Standardization (common parts) ——>- = 
— Vender Market tg 
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Measuring the Impact of 


RD and REQM CMMI 
Process Areas 

THE 

POWER Aldo Dagnino 


TO KNOW. Director of Manufacturing Solutions 


Introduction 
CMMI Appraisal Results 
Methodology to Measure economic Benefits 


Conclusions 


Why do Defects Occur? 


Poorly defined requirements - approx 55% 


Poor design - approx 25% 
Coding issues -approx15% #& 
Other -approx5% © 


Poor specification, scoping, and communication of 
release requirements is the number 1 failure point 
in making quality , customer-centric software 


Patton, R. (2001), Software Testing, SAMS Publishers, USA 


Reduce the effort and cost of defect removal activities 
after system verification by 30% in Manufacturing 
Solutions Department 


The above business objective guided the scoping of 
the CMMI internal SAS appraisal. 
¢ The CMMI appraisal was focused on CMMI Level 2 
process areas 
e After conducting the appraisal, it became evident 
that a primary source of defect generation were the 
RD and REQM process areas 


Chronology of Events 


RD Specific Goals/Practices - Appraisal Results 


Requirements 
Development PA 
Specific Practices 


SG2 
Develop Product 
Requirements 


SG1 SG3 
Analyze and 
Validate 
Requirements 


Develop Customer 
Requirements 


S¥sts See 


REQM Specific Goals/Practices — Appraisal Results 


RD Specific Goals/Practices - Enhanced Process 


S¥ste ee 


REQM Specific Goals/Practices - Enhanced Process 


RD - Before and After 


Requirements 
Development PA 
Specific Practices 


SG2 
Develop Product 
Requirements 


SG1 ee ; 
Develop Customer| Anal lyze ani 
Validate 


Requirements 


Requirements 


REQ 


M - Before 


Requirements 
Management PA 


SG1 
Manage 
Requirements 


and After 


Requirements 
Management PA 


SG1 
Manage 
Requirements 


New RE Process 


Ry fal i“ LAs Rey 


§SaS | Fs. 


MA Process Area 


= Measurement Objective 


¢ To measure the defect removal effort in hours spent by 
software developers to fix defects identified, documented, 
and categorized in the SAS DEFECTS system 


» Measures 


¢ # of defects in each category associated with RD and REQM 


¢ Actual development time associated with fixing each defect. 
Time includes developers’ time and also subsequent tester’s 
time to review the fixes 


¢ Cumulative time that accounts for all defect types across the 
whole project. 


§SaS | Fs. 


MA Process Area 


= Measurement Data Collection 


¢ Defects are identified by test team and documented in the 
DEFECTS database 


¢ Defect types are defined in a collaborative fashion between 
testers and developers 


¢ Time associated with defect removal will be recorded by 
developers in time-sheet database 


¢ Time associated with reviewing defect removal by testers 
will be recorded in time-sheet database 
= Analysis of Measurement Data 


e Measurement data will be analyzed using the RE Economic 
Benefit Calculation explained later 


MA Process Area 


= Reporting of Measurement Data 
e Three primary reports will be generated 


— Histogram that presents the number of defects within 
their types at time t1 and at time t2 


—- Cumulative defect removal time graphics for each 
defect type for time t1 and time t2 


— Histogram of total time and cost for all defect removal 
for time t1 and time t2 


Defect Types Associated with Requirements 
Development 


= Market Requirements Document (MRD) 
e Requirement not documented in MRD 
e Requirement not properly specified in MRD 


e Requirement creep in MRD 

¢ Development misunderstood requirement in MRD 
= Product Requirements Specification (PRS) 

e Requirement not documented in PRS 

e Requirement not properly specified in PRS 


Defect Types Associated with Requirements 
Management 


= Change in requirement not documented in MRD 
= Change in requirement not documented in PRS 


= Change in requirement not properly documented in 
MRD 


= Change in requirement not properly documented in 
PRS 


S¥ste ae 


Compare Defect Removal Effort and Cost 


a End of Beginning Beginning End of Q1_ —s Beginning 
Timeline Q1 2006 Q2 2006 of Q3 2006 2007 Q2 2007 
Process 
RE * (MRD-PRS- 
t 
Process sel => Change Memi) —SS 
fe 
Defect 
Removal and 
SWA 3.6 Track Time Release 
Employed 
er Defect 
SWA 4.0 eritication Removal and 
SWA 4.0 MRD PRS Development and Defect Track Time 
Classification Employed 


Compare Effort and 
Cost of Defect Removal 
between SWA 3.6 and 
SWA 4.0 


RE Economic Benefit Calculation 


t 1 = time to fix defects of type “k” at time t, 


t >) = time to fix defects of type “k” at time t, 
t 4 «i = time to fix defect of type “k” at time t, and at occurrence | 
t > «qj = time to fix defect of type “k” at time t, and at occurrence | 
C = hourly average cost of development 


T,= time differential between fixing all defect types at 
time 1 and at time 2 


n 
tiw= 2, [ti«i] 
m 
Com = an [tray] 
Ss 
Ta = 2 it ay -~ter] 


Expected Economic Benefit = Ta+C 


Conclusions 


=" Having a quantitative business objective guiding the 
internal CMMI appraisal and process improvement 
activity is essential 


= Use the Measurement and Analysis PA as a basis 
for measurement 


= Start “small” 


= Find “high impact” area that shows a “large” benefit 


Untethered Activities — the Real 
Reason for Schedule Slips 


NDIA CMMI Technology Conference 
Thursday, November 16th 
Rolf W. Reitzig 
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Schedule Mismanagement Statistics 


The average project runs 
189% over budget and 
overshoots its schedule 
by 222%! 

More than 72% of IT 
projects fail to deliver 
expected results? 


The completion rate of IT 
projects lasting a year or 
more is only 51%? 


1: The CHAOS Studies, The Standish Group International, Inc., Dennis, MA, 1994-2002 
2: The Hackett Group, Atlanta, GA, 2002 
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What Causes Schedule Misses? 


Senior Management Mandate? 
Scope Changes? 
Poor Estimation? 


Poor Planning/Untethered 
Activities? 
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Senior Management Mandate? 

¢ Inthe absence of credible planning data and schedules, 
management will set targets for teams to meet 

« There are some managers that believe that stretch 
goals are a good way to motivate teams 

¢ Other times, mandates reflect other business problems 


— Competitors have announced a competing product launch, 
therefore we have to match them 


— Executives making commitments to customers for delivery 
dates — the Bat Phone! 
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Scope Changes? 
¢ Too often, projects are asked to 
“suck up” unplanned scope changes 
— “Our competitor just announced this 
feature — we have to have it in the next 


release!” 
— “| know | didn’t tell you about this 
critical piece of functionality, but | need 
it anyway!” 
¢ “Death by a thousand cuts” 
— Numerous “small” requirements/scope 
changes eventually add up to real work 
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Poor Estimation? 


To help themselves, project teams attempt to adopt 


Slide 5 


elaborate estimation techniques and tools to improve 
their project planning and management performance. 


— COCOMO 

— Wide Band Delphi 

— Parametric Tools 

But, this typically doesn’t help!!!?'?!!! 


How can you estimate activities that aren’t in your 


plan? 


cog NENCEinc 
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CMMI Improves Estimation Techniques 


¢ CMMI Specific Practices 


— Project Planning 


* SP 1.1- Establish a top-level work breakdown structure (WBS) to 
estimate the scope of the project 


¢ SP 1.2 —Establish and maintain estimates of the attributes of the 
work products and tasks 


* SP 1.3 — Define the project life-cycle phases upon which to scope 
the planning effort 


* SP 1.4- Estimate the project effort and cost for the work products 
and tasks based on estimation rationale 
— Integrated Project Management 


* SP 1.2 - Use the organizational process assets and measurement 
repository for estimating and planning the project’s activities 
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Actual versus Estimated Effort 


If we add up the effort for 
many software projects, we'd 
find that more actual effort is 
expended than estimated 
Where is this extra effort 
coming from? 

- Poor estimation? 

— Scope changes? 

— Unplanned activities? | 
Are you even estimating Estimated Effort Actual Effort 
and tracking all the effort 
associated with the 
project??? 
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Are Better Estimation Techniques Really The Answer? 


¢ The project team needs to present management with 
credible schedules and data-driven plans 
Need the right “recipe” for developing systems 
— How much of the estimated effort should you allocate to: 
* Requirements Development? 
* Configuration Management? 
* Process & Product Quality Assurance? 
PCs 
— Are you even thinking about your projects in this fashion? 
Once you know all the activities (the process) that go 
into building your systems, you can implement better 
estimation 
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Poor Planning/Untethered Activities? 


Untethered activities are those that must be 
performed, but are never planned or scheduled 


Since these untethered activities must be completed, 
others are short-sheeted or delayed to get the job done 


The end result is a delayed schedule, or worse, a poor 
quality system that required expensive rework and thus, 


more delay 
Start Date Planned End Date Actual End Date 


Unplanned, Additional 
Original Estimated Effort Untethered Testing, 


Activities Rework 
Unplanned, Untethered Activities 
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Post-Release 
Rework! 


Typical (somewhat simplified) CMMI ML 1 WBS 


Task Name Sep 17, '06 Sep 24, '06 | Oct 1, '06 Oct 8, '06 
S(S|M|T [wit [F/S|S|M|T |wIiT[F/S|S[M|T |WIT[F/S|S|M|T |W/T|F[S 


‘-] Plan P 1 
Forget this 
Understand Customer Needs jannind. we have 
Develop the Plan Pp i 9; na 4 
Approved Plan coding to do: 


| Requirements 
Develop the Requirements How am! supposed to 

= Design finish this spec with all 
Develop the Design these project status 

= Build meetings | have to attend? 
Build the Hardware 
Build the Software 
Perform Integration Test 


Darn! I’m not done with 
the design yet. Better 


= System Test i 
cancel the peer review. 


Perform System Test 
Release 


| can’t worry 
Stop finding all about unit 
these bugs! We testing! I’m 
have to release! i 


We can’t deliver the 
product with all 
these problems - 
We need to delay... 
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* A project utilizing iterative/agile techniques used story cards to 
plan and scope the work for a 2 month iteration 
— Cards were to represent 1 day of effort and a specific deliverable 
- eupeyees were instructed that they could not perform work without a 

Car 

By the end of the iteration, the card count “exploded” by 50% 
and the schedule slipped by 1 month, although no new 
functionality was authorized 
Much of the card explosion was due to necessary but . 
untethered activities such as configuration management, builds, 
system administration, test computer installation/configuration, 
etc. 
Very little was due to poor estimation of the product 
development work! 


11/46/2006 cognenceinc Slide 12 


©2006- cognence, inc. Improving Software Economics 


How Can You Tell? 
¢ Look at schedules 


¢ Ask people if everything they do shows up ona 
schedule and whether they track time against it 


¢ Do schedules map to processes? 
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Benefits of Using the CMMI - Maturity Level 1 


pa) 
~ 
a 
— 
2 
ise} 
a) 
oO 
o 
jaw 


Actual 


Time, §$, etc. 


Source: The Capability Maturity Model - Guidelines for Improving the Software Process 
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Benefits of Using the CMMI - Maturity Level 3 


Target 
CMMI ML 1 CMMI ML1 


Target Actual 


Probability 


Time, $, etc. 


Source: The Capability Maturity Model - Guidelines for Improving the Software Process 
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Leading Organization 
CMMI Maturity Level 3+ 
% Effort Distribution (Hours) 


Average Organization 
CMMI Maturity Level 1 


Far less of the ature 
% Effort Distribution (Hours) 


organization is 
fixing problems 


So, a larger 
percentage can 
develop new 
solutions! 


B Design @ Code +UT 


@ Reqs @ Design @ Cod @ Reqs 
@ Quality Test 


PM Quality Test oPM 
@ Rework @ Rework 


¥ Organizations with mature development process infrastructures plan for more requirements, 
design, code reviews and other quality activities (PPQA) to reduce testing/rework later 


v Resources focused where they need to be — new releases/innovation! 
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CMMI/SCAMPI Helps 


Po Engineering ess Managemen 
ac 
[SSSeel Geeeed Goce) Emme ieee 
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CMMI Focuses on Project Management First 


¢ Maturity Level 2 
Project Management Process Area 
Project Monitoring & Control 
Supplier Agreement Management 
Measurement & Analysis 
Requirements Management 
- Although M&A and RM are not PM process areas, some would argue 
managing scope and measuring are project management 
responsibilities 
¢ Clearly, the CMMI believes better Project Management is a key 
enabler to better performance! 


¢ So, is better estimation really the key? 
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How Does CMMI Address Untethered Activities? 


* Generic Practices 
GP 2.2 - Establish and maintain the plan for performing the process 


GP 2.3 - Provide adequate resources for performing the process, developing the work 
products, and providing the services of the process 


GP 2.4 - Assign responsibility and authority for performing the process, developing 
the work products, and providing the services of the process 


GP 2.7 - Identify and involve the relevant stakeholders of the process as planned 
GP 2.8 - Monitor and control the process against the plan for performing the process 
and take appropriate corrective action 
Only four specific practices at ML 2 related to estimation, and one more at 
ML 3 
There are 45 generic practices at ML 3 related to planning and 
addressing untethered activities, and another 30 related to tracking! 
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How Can We Eliminate Untethered Activities? 


Stop senior management mandates and allow project teams to 
develop, defend, and negotiate schedules 

Develop standard WBS templates based on organizational 
processes that account for all the activities projects should 
undertake 


Ensure project schedules/plans account for all activities 
— No untethered activities! 


— Ensure quality activities are clearly called out — reviews, audits, CM, 
etc. 


— Provide placeholders for requirements changes, risk management 
activities 
¢ Implement an activity-based time reporting/cost accounting 
system 
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10 


Many people attempt to establish elaborate estimation 
techniques to fix schedule problems 


The real issue (initially) is not the estimation technique 

Rather, incorporating all the activities that will be executed in 
plans and schedules is key — eliminate untethered activities! 
Better planning also helps to combat senior management 
mandates and unplanned scope changes 


If you aren’t estimating and tracking all effort associated with 
a project, you’re shooting in the dark 
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> 
accenture Purpose & Agenda 


Instead: 


e Balance internal and external tools/techniques, one of which can 
be CMMI 


e Make sure your business goals are the focus 


e Understand when CMMI adds value and when you need 
something else (instead of or in addition to) 


Copyright © 2006 Accenture All Rights Reserved. 


= Context 
accenture — What is Accenture? 


¢ US $15.5 billion net revenues for FYO5 
¢ Over 133,000 employees in 48 countries 


e Accenture serves approximately 2,500 clients that span the full 
range of industries around the world 


e We serve 87 of the Fortune Global 100, two-thirds of the Fortune 
Global 500 and government agencies in 26 countries 


¢ Of our top 100 clients in fiscal 2005, 96 have been clients for at 
least five years and 83 have been clients for at least 10 years 


e Ranked No. 1 in "Computer & Data Services" industry category in 
Fortune's 2005 list of America's most admired companies 


¢ InformationWeek ranked Accenture as the #1 leading Outsourcer 
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* Delivery Excellence 
accenture — Holistic Approach to Quality 7 


e From a quality model perspective, you need to balance internal 
and external definitions of quality and tools/techniques 


e Internal: proprietary best practices help differentiate you and 
build unique strengths 


— Methods — Stakeholder Management 
— Tools — Continuous Improvement 
— Metrics — Quality Assurance 


Architectures 
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. Delivery Excellence’ | 
accenture —_ Holistic Approach to Quality // 


eSCM-SP: focus on IT- 


e External: industry quality enabled sourcing 
models provide an objective | | 
view into best practices anda [ea 
common yardstick to compare Quality Management Syste: 


companies 

— Industry standards frequently 
overlap, but have different Systems aos 
‘sweet spots’ Integration BPO re 


ITIL: focus on IT 
Service Management 
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& Delivery Excellence 
accenture = Case in Point: CMMI 


¢ Cost: 4.5 percent decline in overhead and up to 33 percent 
decrease in the cost to fix defects 

¢ Schedule: 30 percent increase in productivity, 50 percent 
reduction in release time, increase of milestones met from 50 
percent to 95 percent 

¢ Quality: reduction in defects found from 6.6 to 2.1 per thousand 
lines of code (KLOC) 

« Customer Satisfaction: increase in client satisfaction scores 
and award fees 


¢ Return on Investment: 5:1 to 13:1 ROI 


From CMU/SEI-2003-SR-009 Demonstrating the Impact and Benefits of CMMI®: An Update and Preliminary Results. 
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_ Delivery Excellence 
accenture = Case in Point: CMMI 


Common pitfalls: 
e Setting wrong or incomplete goals 
¢ Believing CMMI will fix all or most of your problems 
e Focusing your improvement program on CMMI by itself 


From CMU/SEI-2003-SR-009 Demonstrating the Impact and Benefits of CMMI®: An Update and Preliminary Results. 
Copyright © 2006 Accenture All Rights Reserved. 


* Delivery Excellence 
accenture —_ Goal Setting 


¢ Do not confuse CMMI goals with your business goals 


e What are your business goals? Some examples: 
— Increase customer satisfaction 
— Reduce delivery time 
— Increase productivity 


e How can CMMI help you with your goals? 


— No: Customer satisfaction is low because our call center 
is understaffed and keeps them on hold too long 

— Maybe: Customer satisfaction is low because our system 
upgrades take too long and have lots of bugs 
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- Delivery Excellence 
accenture —_ Goal Setting 


¢ How do we know when we’ve achieved success? 
Which of these is more successful: 


— We are CMMI Level 3 but customer satisfaction hasn't 
changed 


— Customer satisfaction has improved 40% but we ‘failed’ 
our CMMI appraisal 


¢ Common mistake: defining success based on CMMI Level 
rather than achievement of business goals 
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& Delivery Excellence 
accenture = CMMI isn’t everything 


¢ CMMI has some excellent content, such as: 
— Project management 
— Systems development/maintenance 


e What about other problems related to your business goals? 
— Customer relationship management 
— Human performance 
— Outsourcing 
— Infrastructure management 
Security management 
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> Delivery Excellence 7% 
accenture = CMMI isn't everything (Ties 


pay On eee 


e Make sure to supplement your improvement program with other 
quality frameworks that have different ‘sweet spots’ 


People Capability Maturity Model (P-CMM®) 

eSourcing Capability Model for Service Providers (e@SCM-SP) 
or for Client Organizations (eSCM-CL) 

Information Technology Infrastructure Library (ITIL®) 

ISO® 9000 

(Lean) Six Sigma 

ISO 17799/27001 


Copyright © 2006 Accenture All Rights Reserved. 11 


- Delivery Excellence 
accenture CMMI isn’t everything 


¢ CMMI has some excellent ¢ But don’t forget: 
content and principles to help — Communications program 
structure an improvement — Human performance 
rogram 
Prog — People change 
— Sponsorship management 
— Training — Tool and technology 
— Measurements improvements 
— Documentation — Organizational 


— Verification re-engineering 
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: Delivery Excellence 
{Olt 100 | doe @Lo)alerlUTSlelals 


¢ Focus first on business strategy and objectives 


¢ Decide which quality frameworks will best help with those 
— Internal, proprietary methods and frameworks 
— Industry standards: CMMI, ITIL, eSCM-SP, Six Sigma, etc. 


¢ Define your improvement program 
— Business case 
— Program plan + milestones 


— Define ‘success’ and identify metrics that will help you measure 
SUCCeSS 
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: Delivery Excellence 
accenture = Conclusions 


e Make sure your improvement program has all the elements, 


including: 
— Program Management 
— Sponsorship 
— Process, Technology, and People Change Management 
e Training ¢ Tools 
¢ Communications e Process and Organization 


re-engineering 
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Keith M. Heston 
Certification Program Manager, Accenture 


keith.m.heston@accenture.com or kheston@cs.cmu.edu 
+1 412 809 5831 


See http://www.accenture.com 
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Resolving CMMI Issues 


CMMI Technology Conference 
16 November 2006 


Dr. Karen J. Richter 


IDA Summary Issues 


> Programs execute at lower maturity levels than their 
organizations have achieved and advertised 


> High-maturity practices are not consistently applied at 
the project level after contract award 


> New projects do not always incorporate the 
organization’s CMMI-appraised processes 


> Content of Appraisal Disclosure Statements (ADS) is 
lacking 


>» Appraisal sampling procedures do not ensure adequate 
coverage of the organizational unit 


> Appraiser quality not always consistent— training, etc. 
> Lack of agreement on what constitutes Levels 4 and 5 
> Need to converge to a single representation 

> Inadequate training and education for acquirers 

> Should CMMI be used for source selection? 


= 


A Program Execution 


> Issue 


«Programs execute at lower maturity levels than their 
organizations have achieved and advertised 


> Resolution 


“* Defense Contract Management Command (DCMA) 
has begun data collection on program performance 


Project Implementation of Appraised 
IDA Processes 


> Issues 
“* High-maturity practices are not consistently applied 
at the project level after contract award 


«* New projects do not always incorporate the 

organization’s CMMI-appraised processes 
> Resolution for v. 1.2 

“* Eliminated “level for life’-—now a 3 year limit 

«* Added a specific goal and two specific practices to 
the Organizational Process Focus (OPF) process 
area (PA) of the model to stress deployment of 
processes to projects 

«* Added words to a specific practice in Integrated 
Project Management (IPM) PA to emphasize having 
a defined process at project start-up 


[DA __ Appraisal Disclosure Statement (ADS) 


> Issue 
** Content of Appraisal Disclosure Statement (ADS) is lacking 


> Resolution for v. 1.2 
** Much more detail disclosed in v. 1.2 ADS than previous 
versions 


¢ Organizational unit, focus and non-focus projects, application 
domains 


** Lead Appraisers must certify that focus and non-focus projects 
are truly representative 


** Lead Appraisers must certify that level 4 and 5 appraised 
Subprocesses map to organization’s business objectives 


“* All appraisals performed after 28 Nov 2006 must report with v. 
1.2 of the ADS, regardless of model or method used 


** DoD contractor’s ADS will be posted on a .mil web site 


IDA Sampling 


> Issue 


«* Appraisal sampling procedures do not ensure 
adequate coverage of the organizational unit 


> Resolution for v. 1.2 


«* New sampling rules and disclosure in Appraisal 
Disclosure Statement (ADS) 
¢ Organizational Scope: name, type, location 


¢ Organizational Unit Coverage: size, application domain, 
geographical breadth, project type expressed in percentages 
of total organizational unit 


> Issue 
«* Appraiser quality not always consistent-— training, etc. 


> Resolution for v. 1.2 
“+ Upgrade face-to-face training with focus on integrity 
¢ Sessions held at this conference 
“* Certification of high-maturity appraisers in place 


¢ Workshop on high maturity training determined the 
requirements for which the lead appraisers have to provide 
evidence in order to do high maturity appraisals 
—Previous courses, oral questions, etc 


IDA Levels 4 and 5 
> Issue 
“*Lack of agreement on what constitutes Levels 4 


and 5 


> Resolution for v. 1.2 


«* Body of Knowledge on high maturity will be 
developed and available by October 2007 
¢ What constitutes high maturity behavior? 


Addressing Acquisition 


= 
x 


> Issues 
** Inadequate training and education for acquirers 
** Should CMMI be used for source selection? 


> Resolutions for v. 1.2 


** Completion of Guidebook for Program Managers 
e Understanding and Leveraging Your Contractor’s CMMI Efforts: A 
Guidebook for Acquirers 
¢ Due out after the first of the year—February 2007 
** Development of the CMMI for Acquisition Constellation (CMMI- 
ACQ) 
e Initial draft model developed by General Motors and the SEI 
e Involvement of broad acquisition stakeholder community through 
pilots and as key reviewers 
¢ CMMI-ACQ model and training to be available by end of April 2007 


Summary Issues—Addressed 
IDA in V. 1.2 


v Programs execute at lower maturity levels than their 
organizations have achieved and advertised 


v High-maturity practices are not consistently applied at 
the project level after contract award 


v New projects do not always incorporate the 
organization’s CMMI-appraised processes 


Y Content of Appraisal Disclosure Statements (ADS) is 
lacking 


v Appraisal sampling procedures do not ensure adequate 
coverage of the organizational unit 


VY Appraiser quality not always consistent-— training, etc. 
Vv Lack of agreement on what constitutes Levels 4 and 5 
> Need to converge to a single representation 

vY Inadequate training and education for acquirers 

¥ Should CMMI be used for source selection? 


A Outstanding Issues 


= 


> Need to converge to a single representation 


> Issues addressed in v. 1.2 product suite still 
need to be monitored to ensure improvement 
goals are really being met 
“Integrity of appraisals 
“* Quality of the product suite 
“* Education of acquirers 
“* Opportunities for streamlining where appropriate 


A Moving Forward 


= 


> Ensure constellations don’t become stovepipes 
> Monitor outstanding issues—DCMA continues 
to investigate 


«Programs execute at lower maturity levels than their 
organizations have achieved and advertised 


> Continue to address 


“*Lack of agreement on what constitutes Levels 4 
and 5 


«* Need to converge to a single representation 


7y DA Additional Improvement Opportunities 


> Lean the Model 
«* Convene 6-sigma team to analyze 


“* Eliminate cumbersome material included for legacy 
reasons 

«* Move forward from legacy representation approach 
to focus on measuring process capability with 
translation to organizational maturity 


> Next Generation Process Improvement (PI) 


«*Develop a strategy for how we will take PI to the 
next level 

“*Do we need something else with CMMI for high 
maturity? 
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Questions/Comments? 


Why Choose Between ISO, CMMI, 


ITIL and Six Sigma? 
Why not Leverage All! 


Lemis O. Altan 


— 
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A Few Questions 


¢ |SO, CMMI, ITIL, Six Sigma. How many of you have 
wondered which model or which standard to choose from? 


¢ How many of you have heard of or participated in 
conversations/discussions on which Is better? 


¢ How many of you wondered how all these models, standards 
might fit together and you might actually use them all in your 
organization? 
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Session Objectives 


¢ Review of ISO 9001:2000, CMMI, ITIL and Six Sigma 
¢ Present the relationships among them 


¢ Discuss the areas where each would be applicable in your 
organization 
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Typical Organizational Structure 


Executive Offices 


Engineering] (FF [HA] [Finance] [Purchasing 


biaraiiy 
Services 
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Motives for Process Improvement 


e Internal drivers to improve product quality and reduce cost 


¢ External drivers to achieve a certification or rating for 
marketing purposes 
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ISO 9001:2000 Standard 


4/13/2006 


Applies to all organizations that develop products or provide 
services 


Equally applicable to small, medium, and large organizations 
Less prescriptive than the 1994 version 


Provides an organization with more flexibility in documenting its 
Quality Management System (QMS) 


Enables each individual organization to develop the minimum 
amount of documentation needed in order to demonstrate the 
effective planning, operation and control of its processes, and the 


implementation and continuous improvement of the effectiveness 
of its Quality Management System 
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ISO 9001:2000 Requirements 


¢ Systemic 


Establish quality system 
Document quality system 


¢ Management 


Support quality 

Satisfy customers 

Establish a quality policy 
Carry out quality planning 
Control quality system 
Perform management reviews 


e Resource 
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Provide quality resources 
Provide quality personnel 
Provide infrastructure 
Provide quality environment 


¢ Realization 


Control realization planning 
Control customer processes 
Control product development 
Control purchasing function 
Control operational activities 
Control monitoring devices 


¢ Remedial 
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Perform remedial processes 
Monitor and measure quality 
Control nonconforming products 
Analyze quality information 
Make quality improvement 
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ISO 9001:2000 Requirements 


Quality management system 

Quality manual 

Human resources 

Resource management 

Competence, awareness and training 
Infrastructure 

Work environment 

Meet customer requirements 

Customer communication 

Identify customer requirements 

Plan, design and development 

Design and development verification 

Design and development validation 

Carry out design and development 
Monitoring and measurement of process and product 
Internal audits 

Control of design and development changes 
Control of monitoring and measuring devices 
Control of documents and records 
Improvement 

Verify purchased products 


4/13/2006 Cognenceinc 


©2006- cognence, inc. Improving Software Economics 


Slide 8 


Capability Maturity Model Integration (CMMI®) 


4/13/2006 


A framework consisting of best practices that address product 
development and maintenance 


V1.1 emphasizes systems and software engineering and the 
integration necessary to build and maintain the total product 


Applicable to any size engineering organization 
Much more descriptive then ISO — Specific Practices 
Tells you what to do, not how to do it 

Has several bodies of knowledge 
Two representations: 

— Continuous, Staged 
Assure consistency with ISO/IEC 15504 
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CMMI Continuous Representation 


CATEGORY PROCESS AREA (V1.1) 

Process Organizational Process Focus (OPF) 

Management Organizational Process Definition (OPD) 
Organizational Training (OT) 
Organizational Process Performance (OPP) 
Organizational Innovation and Deployment (OID) 

Project Project Planning (PP) 

Management Project Monitoring and Control (PMC) 


Supplier Agreement Management (SAM) 
Integrated Project Management (IPM) 
Risk Management (RISKM) 

Integrated Teaming (IT) 

Integrated Supplier Management (ISM) 
Quantitative Project Management (QPM) 


Engineering 


Requirements Management (REQM) 
Requirements Development (RD) 
Technical Solution (TS) 

Product Integration (Pl) 

Verification (VER) 

Validation (VAL) 


Support 
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Configuration Management (CM) 

Process and Product Quality Assurance (PPQA) 
Measurement and Analysis (MA) 

Organizational Environment for Integration (OEI) 
Decision Analysis and Resolution (DAR) 

Causal Analysis & Resolution (CAR) 
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© CMU, Software Engineering Institute 
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ISO 9001:2000 and CMMI V1.1 Overlap 


Quality management svstem - OPF, GGs 
Quality manual- OPD, GG 

Human resources 

Resource management - PMC 

Competence, awareness and training - OT 
Infrastructure - OEl 

Work environment- og 

Meet customer requirements - REQM, VAL 
Customer communication - PMC 

Identify customer requirements - RD 

Plan design and development- PP 

Design and development verification - VER 
Design and development validation - VAL 
Carry out design and development - TS 
Monitoring and measurement of process and product - MA 
Internal audits - PPQA 

Control of design and development changes- CM 
Control of monitoring and measuring devices 
Control of documents & records- PPQA 
Improvement- OPP CAR 

Verify purchased products- SAM 
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a 


OID 
IPM 
IT, RISKM 
ISM 
QPM 


PI, Walk, WA 


TT OP 


DAR, CAR 
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ISO 9001:2000 and CMMI V1.1 Overlap 


Monitoring and 
measurement of 
QPM process, product system 


Plan design & OPF Quality OPD 


development manual 


I OT 
Design and Competence, _ PPQOA audit S/ sing 
psycopment awareness and training Control of records 


PMC 


Customer communication 


IPM OPP Resource management ; 
improvement g Control of design and 


Infrastructure 


aeperenent Work environment VER 


VAL 


D 
Customer-related Design and de ent validation 
RE 
Meet customer 


ments 


ISM 


measuring devices 


4/13/2006 cognenceinc © CSM, no Slide 12 


©2006- cognence, inc. Improving Software Economics 


ISO 9001:2000 and CMMI Summary 


Considerable overlap 


ISO has a broad perspective and is more general 


CMMI provides more detail on what is required for engineering 
best practices 


ISO is applicable to all types and sizes of organizations 
CMMI Is applicable to the Engineering function, IT 
application development function, Professional Services 
development function and any other organization that is 
planning and managing projects. 
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Information Technology Infrastructure Library (ITIL) 


4/13/2006 


©2006- cognence. 


Premise: 


— IT plays a critical role in support of the business goals and 
requirements 


— IT provides essential services to the organization supporting the 
business 


Set of integrated, best practices focused on the management of IT service 
processes 


Promotes quality and efficiency in the use of Information Technology 
Meets ISO 9001 standards 
Considered "public domain" because it is a Crown Copyright 


Focuses on IT Service Management - IT Service Support and IT Service 
Delivery (48 modules) 


ITIL instructs how to do it 
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ITIL Service Management Focus -1 


IT Service Support 


4/13/2006 


Configuration Management - physical and logical perspective of 
the IT infrastructure and the IT services being provided 


Change Management - standard methods and procedures for 
effective managing of all changes 


Release Management - testing, verification, and release of changes 
to the IT environment 


Incident Management - the day-to-day process that restores normal 
acceptable service with a minimal impact on business 


Problem Management - the diagnosis of the root causes of incidents 
in an effort to proactively eliminate and manage them 


Service Desk (Function) - a function not a process. This provides a 
central point of contact between users and IT 
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ITIL Services Management Focus -2 


IT Service Delivery 
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Availability Management - optimizing IT infrastructure capabilities, 
services, and support to minimize service outages and providing 
sustained levels of service to meet business requirements 


IT Service Continuity - managing an organization's capability to 
provide the necessary level of service following an interruption of 
service 


Capacity Management - enabling an organization to tactically 
manage resources and strategically plan for future resource 
requirements 


Service Level Management - maintaining and improving the level of 
service to the organization 


Financial Management for IT Services - managing the costs 
associated with providing the organization with the resources needed to 
meet requirements 
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CMMI V1.1 and ITIL Touch Points 


CATEGORY 


Project Mgt. 


PROCESS AREA 
PP - Problem Mgt., Change Mgt., SW Control and Distribution, Computer Network Operations, 
Capacity Mgt., Security Mgt., Contingency Planning, Planning and Control 


PMC - Problem Mgt., Change Mgt., Contingency Planning, Planning and Control, Business IT 
Alignment 


IPM - Customer Relationship Mgt., Service Level Mgt., Planning and Control, Business IT 
Alignment 


RISKM - Change Mgt. , Contingency Planning, Planning and Control 
IT - Service Level Mgt., Planning and Control 
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REQM - Help Desk, Incident Mgt., Problem Mgt., Change Mgt., Customer Relationship Mogt., 
Service Level Mgt., Availability Mgt., Contingency Planning, Business IT Alignment 


RD - Customer Relationship Mgt., Service Level Mgt., Availability Mgt., Business IT Alignment 
TS - SW Control and Distribution, Computer Network Operations 


PI - Help Desk, Incident Mgt., Problem Mgt., Change Mgt., Contingency Planning, Planning and 
Control 


VAL - Customer Relationship Mgt,, Service Level Mgt., Capacity Mgt. , Business IT Alignment 


CM - Help Desk Incident Mgt., Problem Mgt., Change Mgt., Configuration Mgt., SW Control and 
Distribution, Computer Network Operations, Security Mgt. 


MA - Problem Mgt. 


Ccognenceinc Slide 17 


Improving Software Economics 


CMMI V1.1 Process Areas with No ITIL Touch Points 


¢ Organizational: 
— OPF, OPD, OT, OPP, OID 
¢ Project Management: 
— SAM, ISM, QPM 
e Engineering: 
— VER 
¢ Support: 
— PPQA, DAR, CAR, OEI 
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CMMI V1.1 and ITIL Touch Points 


cyt pce Mgt. 


- Mgt.: Problenx here, Capacity, Security 
- SW Controldnd Distribution, 
~ Network Ops., ~ SW Control and Distribution, 


- Contingéncy Planning, 
— Planni i ane Foca TS - Computer Network Operations 


Mgt.: Help Desk, Problem, Change, Contingency Planning, Rilanning and Control 
- Mgt.: Customer H@bationship, : g gency g g 


Service Level, PMC 
- Planning and Control -Mgt.: Problem, Change cry - Mgt.: Help Desk, Incident, _ 
- Bus. IT Align. IPM - Contingency Planning Sieatlly Change, Configuration, 
— Planning and Control ecurity, 
RISKM ~ Change Mgt - Bus. IT Aen. - SW Contro/and Distribution 


- Contingency Planning 
- Planning and Control 
- Service\_evel Mot. - Mgt.: Customer — Mgt.: Customer Relationship, 
IT - Planning and Control ae Relationship, VAL  Servieé Level, Capacity, 


Service Level, REQM ~5#s.!T Align. 


— Network Qps. 


YN - Mgt:Help Desk Incident, Problem, 
Availability, change, Customer Relationship, 
- Bus. IT Align. Service Level, Availability, 
Contingency Planning 
- Bus. IT Align. 
ange = sti(<‘é;O.!!O. vl lCOgNONCRIncdLULULUCUC™~”~COGSlide tl 
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ISO 9001:2000 , CMMI Overlap and ITIL Touch Points 


Quality management system OPF , GGs 

Quality manual OPD, GG 

Human resources 

Resource management PMC - Mgt.: Problem, Change, Contingency Planning, Planning and Control, 


Competence, awareness and training OT Business IT Alignment (Bus. IT Align.) 
Infrastructure OEl 


Work environment OEI 

Meet customer requirements REQM- Mgt.: Help Desk, Problem, Change, Customer Relationship, 

Customer communication PMC Service Level, Availability, Contingency Planning, Bus. IT Align. 
Identify customer requirements RD- Mgt.: Customer Relationship, Service Level, Availability, Bus. IT Align. 
Plan design and development pp-Mgt.: Problem, Change, Capacity, Security, SW Control and Distribution, 
Design and development verification VER Network Ops., Contingency Planning, Planning and Control 
Design and development validation WAL - Mgt,: Customer Relationship, Service Level, Capacity, Bus. IT Align. 
Carry out design and development TS - Software Control and Distribution, Computer Network Operations 
Monitoring and measurement of process and product MA - Problem Mgt. 

Internal audits PPQA 


Control of design ana development changes CM - Mgt.: Help Desk, Incident, Problem, Change, Configuration, 


Control of monitoring and measuring devices Security, SW Control and Distribution, Network Ops. 
Control of documents & records PPQA 


Improvement OPP, CAR 
Verify purchased products SAM 
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CMMI V1.1 and ITIL Touch Points 


yi 0g 
G Problem Mgt. > [0, 
PP ; ‘<0, 
- Mgt.: Problemy Change, Capacity, Security 2 
- SW Control And Distribution, » 
a ee 4 Planning’ TS~ Software Control anu Lisiibution, 


- Planning and Contre - Computer Network Operations 


; vigt.: Hel Desk, Frublem, Change, Contingency Planning, P!anning and Control 
- Mgt.: Custonier elationship, | 


Service Level, PRC. 
Planning anu STB’ - Mgt.: Problem, Change CM-\Nist.: Help Peck Incident, 
Bus. IT Align. - Contingency Panning Pronlem, Change, Configuration, 
RSK - Planning and Control Security, ff 
- Changk WNigi - Bus. IT Align - OM Comrss/and Cistryoution 
- Contingency Planning - Natriark Ys. 
IT 
~ Service evel Mgt: RD - Mat.: Customer WAL- Mat. Custemer Rejationship, 
- Planning anc\Contro Relztionship, REOM vice meee p vacity, 
Service Level, eT. Desk. B ; Ch 
Avallabilit Sai ener fonship, 7 
- Bus. IT Altgit: ervice Level, Availability, 
Contingency, 
- Bus. IT Align. 
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ISO 9001:2000, CMMI V1.1, and ITIL Summary 


¢ Considerable overlap 


¢ 1SO has a broad perspective and is more general 
¢ CMMIL instructs what is required for development best practices 
¢ ITIL provides more detail on the “how to” of a process 


¢ SO is applicable to all types and all sizes of organizations 

¢ CMMI Is applicable to the Engineering function, IT application 
development function, Professional Services development 
function and any other organization that is planning and 
managing projects. 

e ITIL is applicable to IT function, and IT Professional Services 
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Has been described as: 


e “A highly technical method used by engineers and statisticians to fine-tune 
products and processes” 


¢ A goal of near-perfection in meeting customer requirements. 


¢ Asweeping “culture change” effort to position a company for greater customer 
Satisfaction profitability and competitiveness 


SIX SIGMA: “A comprehensive system for achieving, sustaining and 
maximizing business success. Six Sigma is uniquely driven by 
close understanding of customer needs, disciplined use of facts, 
data, and statistical analysis and diligent attention to managing, 
improving, and reinventing business processes.” - six sigma Way, © 2000, 


Peter S. Pande, Robert P. Neuman, Roland R. Cavanaugh 
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¢ Isa flexible system for improved business leadership and 
performance 


¢ Builds on many of the ideas and best practices of the past 
¢ Is about action, not theory 

¢ Applicable to all types and all sizes of organizations 

¢ Could be applied to any process 

e Assumes existing processes 

¢ Consists of six themes: 


— Focus on the Customer — Proactive Management 

— Data- and Fact-Driven Management — Boundaryless Collaboration 

— Process Focus, Management, — Drive to perfection; Lack of Tolerance 
and Improvement for Failure 
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Elements of Six Sigma in CMMI V1.1 


LEVEL 


FOCUS 


PROCESS AREA 


5 Optimizing 


Continuous 
process 
improvement 


e Causal Analysis & Resolution G&D 
e Organizational Innovation and 
Deployment css 


Six Sigma Control 


4 Quantitatively 
Managed 


Quantitative 
management 


° Quantitative Project Management “SSC> ‘SSM 
e Organizational Process Performance ED 


Six Sig 


=| 
> 
s 
3 
< 
© 


3 Defined 


Process 
standardization 


e Organizational Process Focus ©SsID 
e Organizational Process Definition 
e Organizational Training css 


e Integrated Project Management ‘SSC> 
e Decision Analysis and Resolution GD 


e Requirements Development G3) 


2 Managed 


Basic project 
management 


e Requirements Management G2) 


e Project Planning<ssp» 
e Project Monitoring and Control “ssc> 


e Measurement and Analysis 


1 Initial 


Six Sigma Analyze 


Q 


Six Sigma Measure 


u 


Six Sigma Define 


8 
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ISO 9001:2000, CMMI, ITIL and Six Sigma Summary 


¢ Considerable overlap 


¢ |SO has a broad perspective and is more general 

¢ CMMI instruct what is required for engineering best practices 

¢ — ITIL provides more detail on the “how to” steps of the IT best process 
¢ — Six Sigma’s broad perspective could be applicable to any process 


¢ — 1SO is applicable to all types and all sizes of organizations 


¢ CMMI is applicable to the Engineering function, IT application development 
function, Professional Services development function and any other 
organization that is planning and managing projects. 


¢ — ITIL is applicable to IT function and IT Professional Services function 
¢ — Six Sigma Is applicable to any organizational function 
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Questions and Comments 
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cognence Point of Contact 


Ms. Lemis O. Altan 
Candidate — SEI Authorized SCAMPI Lead Appraiser 
Vice President and Principal Consultant 


cognence, inc. 

10101 Grosvenor Place, Suite 411 
North Bethesda, MD 20852 
Telephone: (301) 325-9685 (cell) 
Fax: (303) 731-1616 


Lemis Altan@cognence.com 
www.cognence.com 
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Q) SOEING 


“The Role of CMMI® in Facilitating 
Program Performance in 
Large Scale Systems” 


Dev Banerjee CMMI® Technology Conference 
Director — Systems Engineering and User Group 

Integrated Defense Systems November 13-16, 2006 

The Boeing Company Hyatt Regency Tech Center 


St. Louis, MO Denver, CO 


ING is a trademark of Boeing Management Company. ; 
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Role of CMMI - Boeing View 


SOEINMG 


Forever New Frontiers 


e working ‘oC 
S a global ente 


Strategie 


Run health 
Leverage 
Open ne 


Core competencies 


Detailed customer knowledge and focus 
Large-scale systems integration 
Lean enterprise 


Values 


Leadership 

Integrity 

Quality 

atisfaction 


diverse and involved team 
Good corporate citizenship 
Enhancing shareholder value 


www.boeing.com/vision 


2016 


~ 
—s 
op 
= 
-_ 


Role of CMMI - Boeing View 


"Boeing Progress on CMMI 
= 11 sites at Maturity Level 5 
= Continuing effort to deploy CMMI across all programs 
= Moving to “common” 


«Current Limitations of CMMI 
= Industry Issues 
=» Program Issues 
"Boeing Approach and Initiatives 
= Corporate Initiatives 
= Key Focus Areas — Integrated Defense Systems 
= Program Management Best Practices 
= Strengthen Systems Engineering 


=Summary 
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Role of CMMI - Boeing View 


Yor) ale mrmexeleaiilincsremcemeaynyt 


Programs at 11 major Boeing sites supporting all Business 
Units have been appraised at Maturity Level 5 


Boeing Sites Appraised at ML5 : ' 
Major Programs Appraised: 


12 
¢ B-1B 
10 e C-17 
— ¢ F/A-18 E/F 
v ° F-22 
- 6 ¢ V-22 
r= ¢ Satellite Systems 
g 4) ¢ Mission Planning 
2 e International Space Station 
¢ Delta 
: ¢ GPS 


Dec-03 Dec-04 Dec-05 
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Role of CMMI - Boeing View 
Continue to Deploy CMMI Across All Businesses 


Vale ed cole leclaa 


= Business Units and Sites continue to expand deployment of 
CMMI-based processes 


=" Goals include: 
= Increase the number of programs and projects 
= Increase the penetration into the Engineering population 


= Expand use of CMMI processes from main business 


programs into supporting Engineering activities and 
functions: 


= Modeling and Simulation 
= Engineering Laboratories 
= Ground Support Equipment 
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Role of CMMI - Boeing View 
Move to Common Processes Consistent With 


CMMI Process Areas 


Process Breakdown @ Policies 
Structure ” rf - @ Procedures 
a @ ? rf aa ——— Instructions 


[@)) 

& 

Do 

2 S Process assets (methods, 
as aw ea, wa @Y © process instructions, 

yn 0 templates, etc) 

Q 


Program Unique 
Processes 


Common Processes 


Cross-functional 
process threads 


Legend 

@ Common process 
Program unique 

© used at multiple sites 
Program unique 

© used at single site 


Program Controlled 


®vValue Stream, Business Scenario, Process Thread> 


Mission py ints “ni Fi 
See Concept Concept ([Preliminar Detail Fl 
eee BEG ni ere Maiititen Darciooniesnt | aimee |Past) Gledueuen se ds 
MCR ACR SRR SDR PDR CDR FRR ORR4> DR4 


/PRR 


Program Lifecycle 
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Role of CMMI - Boeing View 


Industry Issues 


Industry Issues: Typical Program Issues:* 
= Top 5 System Engineering Issues = Requirements instability 


NDI = Weak baseline management 


Top | 5 SE a ap 2006 


ractices ki 


aw eenpanink Suelo ep = Weak risk management 
enufirmarics Fs sat gat ary a te Boren mp ie 
ects, Somprann'sng the foundation for 
ite statements tito executable. effective 


Se = Weak supplier management 


anit 1 a and qualit ysti neering expert 
img lent to mer ee yr Ailes np of ihe gov vernmert ai vidi the defe jense 
in 


setoirne tgp ten won tee = Inadequate specifications 


= Poor project management 
=" Top Software Engineering Issues =" Management change control 


Top Software Issues 


‘A pl Late aarp loid requirements upon 
nd managed in ivelopment tS Sustain 


2 Sunder 3st naming econ mace thou * Issues developed during joint Government/Boeing 


3. Solver re life-cycle ots wed facreiaenerd 5 nd 
Suppliers ts inofiective. no cereal 


4. Teens ond gta soternreetnontng Siprtes Systems Engineering Interchange meeting and from 


eet the demands of government and the defense 
industry, 


5) nema siti ict ath list of top reasons projects fail from the Project 


ineffective Nor vrdealing with the je scale onl Compenity 24 rode is 
6. Thera alee arse crc, pit. ale secre Management Institute 


7 inadequate aenion aidecyeie cont etand aioe sues fr 
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Role of CMMI - Boeing View 
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= Boeing Corporate — Growth and Productivity Initiatives 
= Internal Services Productivity 
=» Leant+ 
= Global Sourcing 
= Development Product Excellence 


= Boeing Integrated Defense Systems — Key Focus Areas 
= Supplier Management and Quality 
= Program Management Best Practices 
= Strengthen Systems Engineering 
= Cost Reduction 
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Role of CMMI - Boeing View 


Strong/Robust SE Support of Program Management Best Practices (PMBP) 
Required for Successful Program Performance 


SE Activities Supporting PMBP 


¢e Requirements Definition 
¢ Proposal Risk Management 


i Business Offer ¢ Establish Baseline 
Supplier Integration * Supplier Management 

¢ Product-Focused, 

Organization consistent with WBS 


¢ Performance and 
Operational Measure 
¢ Standard Review Format 


Create and Review ed cole le-laim =<-ree lacey al 
Business Plan FT ale m@xe) ai ige) | 


Risk, Issue and 


Opportunity Mgmt * Trade Studies 


¢ Integration of Risk, 
¢ Requirements Mgmt 


Opportunity, Issues 


Management Process ¢ Technical Performance Measures 
Help Needed and ¢ Affordability 
Independent Reviews - Baseline Management 
pL ¢ Integrated Planning 
¢ IPT, Customer, Teammates, Program * Integrated Scheduling : 
Key Suppliers Involved in evel liiltlalieciitela’ * Allocation/Traceability of Requirements 
Independent Reviews ¢ Integration of Program Baseline 


¢ Interface Definition Management 
¢ Integrated Software Development 
¢ System Integration, V & V 
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Role of CMMI - Boeing View 


Process to Strengthen Systems Engineering 


—_-.__ 


Establish policy/procedure for 


A. Management 
Commitment 


¢ Goals 
: ¢ Program Lead SE 
D. Monitor and 
Support ¢ SE Skills for Program B. Infrastructure 
Leadership 
Program Assessment and Measurement * Program SEMP Establish and Deploy 
* SE Independent Assessments of * SEMP Guide 
“Focus Programs” ¢ SE Leading Indicators/Metrics Guide 


¢ SE Assessment Template 
¢ Proposal/Program Start-up Checklist 
¢ SE Leadership Training 
Program Execution (Scorecard) * “SE Certificate” Program 
« Lead SE on PM Staff ¢ SE Processes, Tools, Training, SMEs 


¢ SEMP ¢ Assessment and Infrastructure for 
Supplier SE 


C. Implementation/ 
Execution 


¢ SE Function Review/Support to 
Proposal/ Program Start-ups 


¢ Progress Measurement/Metrics 


¢ Rollup Reporting to Leadership 


¢ Resources/Structure for SEMP 
Implementation 


¢ SE Reviews and Metrics 


mm Structural Changes * Key Supplier SE Environment 
MEE Operational Changes ¢ Program Self Assessments 
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Role of CMMI - Boeing View 


Summary 


Vision 2016 


Program Supplier Management and Quality 

Management . ; 

Bestprachess Strengthen Systems Engineering Lean + 
Global Sourcing ¥ Development Process 


CMMI Excellence 


¢ Project Planning ¢ Requirements ¢ Configuration ¢ Organizational Process 
* Project Monitoring & Management Management Focus 

Control ¢ Requirements ¢ Process & Product ¢ Organizational Process 
* Supplier Agreement Development Quality Assurance Definition 

Management ¢ Technical Solution »¢ Measurement & * Organizational Training 
¢ Integrated Project - Product Integration Analysis * Organizational Process 

Management+IPPD _ . Verification ¢ Decision Analysis & Performance 
* Risk Management ¢ Validation Resolution ¢ Organizational 

¢ Causal Analysis & Innovation & 


* Quantitative Project 
Management 
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Resolution Deployment 
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DoD Systems and Software Engineering 
Taking it to the Next Level 


CMMI Technology Conference Executive Panel 
14 November 2006 


Mark D. Schaeffer 
Director, Systems and Software Engineering 
Office of the Deputy Under Secretary of Defense (A&T) 


Recent Acquisition & Technology 
Reorganization 


as of June 1, 2006 


Systems and Software Engineering 
Mission Statement 


> Shape acquisition solutions and promote early technical planning 


> Promote the application of sound systems and software engineering, 


developmental test and evaluation, and related technical disciplines 
across the Department's acquisition community and programs 


Raise awareness of the importance of effective systems engineering 
and drive the state-of-the-practice into program planning and 
execution 


Establish policy, guidance, best practices, education, and training in 
collaboration with academia, industry, and government communities 


Provide technical insight to program managers and leadership to 
Support decision making 


Systems Engineering 
Revitalization Effort 


Issued Department-wide Systems Engineering (SE) policy 
Issued guidance on SE, T&E, and SE Plans (SEPs) 


Instituted system-level assessments in support of DAB, OIPT, 
DAES, and in support of programs 


Established SE Forum to ensure senior-level focus within DoD 


Integrating DT&E with SE policy and assessment functions--focused 
on effective, early engagement of both 


Instituting a renewed emphasis on modeling and simulation in 
acquistion 


Working with Defense Acquisition University to revise curricula 
(SPRDE, T&E, PQM, LOG, PM, ACQ, FM, CONT) 


Leveraging close working relationships with industry and academia 


Top 10 Emerging Systemic Issues 


1. Management e IPT roles, responsibilities, authority, poor communication 
Inexperienced staff, lack of technical expertise 


Creep/stability 
e Tangible, measurable, testable 


2. Requirements 


3. Systems Engineering Lack of a rigorous approach, technical expertise 


e Process compliance 


4. Staffing ¢ Inadequate Government program office staff 

5. Reliability e Ambitious growth curves, unrealistic requirements 
e Inadequate “test time” for statistical calculations 

6. Acquisition Strategy ¢ Competing budget priorities, schedule-driven 
¢ Contracting issues, poor technical assumptions 

7. Schedule e Realism, compression 

8. Test Planning ¢ Breadth, depth, resources 

9. Software e Architecture, design/development discipline 


Staffing/skill levels, organizational competency (process) 


10. Maintainability/Logistics * Sustainment costs not fully considered (short-sighted) 
Supportability considerations traded 


Major contributors to poor program performance 5 


Necessary but not sufficient 


Now 


“Take SE to the Next Level” 


Systems and Software Engineering in 
Programs Reduces Costly Mistakes 


; 33% historical RDT&E 
RDT&E Mistakes ee 


Cost Growth 


Under estimating engineering effort is Major source of error 


Applied to 
Sane $222.8B RDT&E FYDP** 


@ Other Mistakes Yields a —__» $73. 52B 
Potential RDT&E Cost Growth FYDP 


OILS Factors; Spares & Support 
SSE 
iMpact g schedule Slips/Management Factors 


DO Engineering/Test/Development 


Percent Cost Growth (CG) 


* SSE positive impact on just 


@ Production Assumptions & Estimation 


1/3 of RDT&E mistakes (11%) 


% of RDT&E Total 


Yieldsa_ $24.51B 


** SAR data for MAIS and MDAP programs under OSD Systems Engineering Oversight 


STRATEGIC 
tc Big A” 


“Little A” 
TACTICAL 


Initiatives For Strategic and Tactical 
Acquisition Excellence 


OBJECTIVES 


Making Decisions that Balance 


the Trade-Space 
¢ Affordable, Feasible Investments 


Starting Programs Right 
¢ Improved, Up-Front Planning 
¢ Awareness of Risk / 

Improved Source Selection 
¢ More Responsive Acquisition 
Solutions 


Process efficiency 
¢ Tailored, agile, transparent 


Program Stability 
¢ No Downstream Surprises 
¢ Issue Awareness 


INITIATIVES 


Portfolio Management 

Tri-Chair Concept Decision / Time- 
Defined Acquisition 

Evaluation of Alternatives 
Synchronize Existing Processes 
Tri-Chair Investment Balance Reviews 


Risk-Based Source Selection 

Small Business Innovative Research 
Acquisition of Services Policy 
Systems Engineering Excellence 
Award Fee and Incentives 


DAB / OIPT Process Optimization 
Common Data / DAMIR 
Restructured DAES 


Program Baseline Assurance 
Capital Accounts 


< ™ Make Decisions that Balance the Trade Space 
' Early Lifecycle Planning 


> Early lifecycle involvement of Systems Engineering: 


e Inform evaluation of alternatives with technical 
insights 


¢ Ensure solutions balance requirements with 
technical feasibility 


e Ensure solutions can be validated and verified 


> Appropriate resourcing (personnel/funding) required 


MSA 


4 T MSB System MSC } 
‘ies & Development & Production & 
y RR* Demonstration Deployment 


*Technology Development and Risk Reduction 


Starting Programs Right 
Shaping Systems Acquisition Solutions 


> System Level 


¢ Application of System Engineering principles 
contributes to successful program execution 


¢ Leverage System Engineering relationship to cost, 
schedule, and performance 


e Ensure enabling disciplines are in concert with 
technical planning 


> Ensuring program and milestone reviews are informed 
by technical planning, verification and validation, and 
complementary business rules 


MSA 


; T MSB System MSC ; 
Ce & Development & Production & 
y RR* Demonstration Deployment 
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Starting Programs Right 


\24 Shaping System-of-Systems Acquisition Solutions 


> System-of-Systems Level 

¢ Needs will be satisfied by groupings of legacy 
systems, new programs, and technologies 

e Presents additional integration and management 
issues 

e Success depends heavily upon software interfaces 

¢ Broad context and knowledge of system 
interrelationships and CONOPS are critical to 
decision-making 

¢ Sound SE practices enable the integration of these 
SoS solutions 


MSA 


Strategic i i T MSB System MSC J 
Planning ¢ goint ; ec & Development & Production & 
Guidance “CONCEPTS y RR* Demonstration Deployment 
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Systems 


Engineering 
Revitalization 


> Competencies Improved 
> Delivered Product Suite 
« Courseware 
¢ Policy/Guidance 
e Program Support 
methods 
> Elevated Stature 
> Raised Awareness 
> Positive Influence 


Vision for Systems Engineering 


and Software 


> 


Systems and Software 
Engineering 
Centers of Excellence 
6S ———S—=— 


World class leadership 

Broaden to Software Engineering, 
System Assurance, Complex 
Systems-of- Systems, Test & 
Evaluation 

Responsive and agile, technical 
discipline to shape acquisition 
solutions 


Ensure appropriate human capital 
needs are met 
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Why the Focus on Software... 


> Research investment has been static or declining 
¢ DARPA computer science R&D funding 50% | (01 - 
04, universities) 
> Requirements growth 10X (% functionality) ’60s -’00s 
> Need vs. skilled/clearable workforce - gaps increasing 
> President’s Information Technology Advisory Committee 
Report, February 2005 
¢ Identifies SW as “major vulnerability” 
¢ Recommends priority attention: “Secure Software 
Engineering and Software Assurance” and “Metrics, 
Benchmarks, and Best Practices” 
> Cost, schedule and performance issues 
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An Organizational Construct 


Director, Systems & 
Software Engineering 


Deputy Director Deputy Director 
Developmental Test Software Engineering & 
& Evaluation System Assurance 


Deputy Director 
Enterprise Development 


Deputy Director 
Assessments & Support 


Management Visibility - Best Practices - Acquisition Excellence 


— Systems and Software Engineering 
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DoD Software Center of Excellence 
Organizational Tenets 


> Support Acquisition Success 


e Ensure effective and efficient software solutions across the 
acquisition spectrum of systems, SoS and capability portfolios 


> Improve the State-of-the-Practice of Software Engineering 


e Advocate and lead software initiatives to improve the state-of- 
the-practices through transition of tools, techniques, etc. 


> Provide Leadership, Outreach and Advocacy 


¢ Implement at Department and National levels, a strategic plan 
for meeting Defense software requirements 


> Foster Software Resources to meet DoD needs 


e Enable the US and global capability to meet Department 
software needs, in an assured and responsive manner 


Promote World-Class Leadership for Defense 
Software Engineering - 


How do we get there? 


Nationa, 


(@slal(slacme)i 
lave liater-valale) 
Excellence 
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\2 4p Systems Engineering Revitalization Framework 


Acquisition Industry 
Community Associations 


SE and T&E Prog ram Academic 
Community 


Support 


Driving Technical Excellence into Programs! 18 


Systems Engineering Policy 


> Policy Memorandum (February 2004) and Policy 
Addendum (October 2004) 


Programs shall apply robust SE approach and 
develop a SE plan 


Each PEO shall have a lead or chief systems 
engineer 


Event-driven technical reviews with entry criteria and 
independent SMEs unless waived by MDA 


OSD shall review program SEPs for ACAT ID and 
IAM programs 


Defense Systems shall establish a SE Forum 
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Systems Engineering Guidance 


> Establish Defense Acquisition Guidebook System Engineering Baseline 
> Developed the following guides: 


Reliability, Availability, and Maintainability 
Integrated Master Plan and Integrated Master Schedule Preparation 
and Use 


Systems Engineering Plans 
Risk Management for DoD Acquisition 
CMMI version 1.2 


> In development: 


Update Defense Acquisition Guidebook 
Contracting for SE Guide 

CMMI Guide for Acquirers 

System Assurance Guide 

Systems-of-Systems System Engineering Guide 
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Systems Engineering 
Education, Training, & Outreach 


Updated /Developed Systems Engineering curriculum 
Reviewed and modified enabling disciplines curriculum 
e Acquisition Program Management, Contract Management, 
Finance, Logistics 
Continuous learning, on-line courses: 
e Developed: Reliability and Maintainability, Technical Reviews, System 
Safety, Modeling and Simulation, Technical Planning 
e In development: Corrosion Prevention and Control, Modular Open 
Systems Approach, Trade Studies 
Established new, strengthened certification requirements for DoD systems 
engineers 
e New Systems Engineering career path provides for broader experience 
and training for selected positions 
Engaging universities: 
Stevens Institute of Technology, University of Southern California, 
Stanford, Southern Methodist, George Mason, Service Academies and 
Naval Postgraduate School, Air Force Institute of Technology Center for 
Systems Engineering 


an 


Driving Technical Rigor Back into Programs 
“Portfolio Challenge” 


> Systems and Software Engineering have been tasked to: 
e Review program’s SE Plan (SEP) and T&E Master Plan (TEMP) 
¢ Conduct program support reviews 


> Portfolio of major acquisition (ACAT ID and IAM) programs, supporting 
10 Domain Areas: 


— Business Systems — Rotary Wing Aircraft 
— Communication Systems — Land Systems 

— C2ISR Systems — Ships 

— Fixed Wing Aircraft — Munitions 


— Unmanned Systems 5 oftwa a Missiles 
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DoD Software Performance: 
What We’re Seeing* 


> Software systemic issues are significant contributors to poor 
program execution 


e Software requirements not well defined, traceable, testable 


e Immature architectures, COTS integration, interoperability, 
obsolescence (electronics/hardware refresh) 


¢« Software development processes not institutionalized, planning 
documents missing or incomplete, reuse strategies inconsistent 


e Software test/evaluation lacking rigor and breadth 
e Schedule realism (compressed, overlapping) 
e Lessons learned not incorporated into successive builds 


¢ Software risks/metrics not well defined, managed 


*Based on ~40 program reviews to date 23 


Other Challenges 


Implementing a DoD vision and strategy for software and systems 
assurance 


Component and Industry adoption and effective implementation of 
sound SE practices as early as possible in the system life cycle 


SE support to Acquisition Initiatives stemming from the QDR 


Retention and development of technical acumen in an aging and 
shrinking acquisition workforce 


Meeting all requests for technical support to programs 
Continue to evolve “high visibility” initiatives: 

- Energy - System-of-Systems 

- CMMI - Modeling & Simulation 


- Defense Safety Oversight Council 
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Global Acquisition 


> Benefits of leveraging reliable non-US sources 
e Price, performance, and schedule benefits 
e Increase interoperability with allies and coalition partners 
e International competition spurs US innovation and efficiencies 
e Mutually beneficial industrial linkages enhance access to global 
markets 
> US is committed to effective systems engineering and best value 
> Norway has demonstrated it is committed to being a Global Supplier 
by: 
e Formation of the National Center of Expertice in Systems 
Engineering 
e Securing significant Joint Strike Fighter work this past year with 
best value products 


Much ts Left to do... Vector is Correct 
25 


How do we get there? 


Nationa, 


(@slal(slacme)i 
lave liater-valale) 
Excellence 
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Systems Engineering Revitalization 


> Policy 
¢ No new policies in 2006 
> Guidance 
¢ Continues to be refined 
> Education and Training 
¢ E&T portfolio refreshed and growing 
> Program Support 


¢ Gaining momentum; yielding systemic insights 
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BETTER YET, YOU 
COULD BUILD THE 
SYSTEM, THEN TLL 
TELL YOUR BOSS THAT 
IT DOESN'T MEET MY 


ILL DESIGN THE 
SYSTEM AS SOON AS 
YOU GIVE ME THE 
USER REQUIREMENTS. 


werw.dilberi.com  scotteder=e@sol.com 


Copyright 93 2883 United Feature Syndicate, inc. 


SP\14 6 3063 Wnttod Feature Gradicete, ine 


Defining User Requirements 


I DON’T MEAN TO 
FRIGHTEN YOU, BUT 
YOULL HAVE TO DO 
SOME ACTUAL WORK. 
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Balancing Key Programmatic Elements 


Systems Test & Risk Acquisition 
Engineering Evaluation Management Exit Criteria Strategy 
; V&V a Mission 
Traceability erie es me Capability 
Organization & Test Resources & 
Technical Test Articles Risk Mitigation Maniga@turing Technical 
Reviews Planning Process 


Focus Areas Technical Technical 
Evaluation Risk Tracking R&M 
Baseline Product 


Linkage w/ 


Other Program pulet Evidence of Enterprise 


Linkage w/ 


erearem Effectiveness Neeaairic Environment 
Mgmt & 


Controls 


Criteria 
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Mgmt & 
Controls 


NORTHROP GRUMMAN 


DEFINING THE FUTURE 


aut What CMMI Does 
e - for Our Industry 
“Sp: = And What It Doesn’t 


3)" V1 ia xed alate) (ole hVar-lale Mm Os-1-) ml Ove) alicla-)alec 
i DY =a hV/:) gum Oxo) (oe) ¢-\o le) 


nd November 14, 2006 


Roger Fujii 

Vice President, Communications and Systems Technologies, 
Defense Mission Systems Division 
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Northrop Grumman Mission Systems & CMMI 


= Strong and continued commitment to CMMI 
= 56 SCAMPI appraisals (39 external) conducted since 8/02 
= 163 projects appraised (115 external) 
= 19 organizational appraisals (67 projects) on PARS Site 
= 14 are Level 5 
= 5 are Level 3 


= NGMS organizations regularly reappraise within 3 years 
= NGMS is committed to using our processes 


= Require all organizations to formally confirm their intent to 
use NGMS processes prior to proposal release 


= Every project undergoes post award process review within 30 
days of start up 


= Process status is a part of every internal program review 


NORTHROP GRUMMAN 
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What CMMI Is Intended to Do? 


#7, = Provide a model of Development best practices 
| = Soon to be expanded to include Acquisition and Services 
= Provide a framework for our organizational processes 


Provide an outline for achieving organizational maturity 
" through exploitation and repetition of those practices 


= Provide informational examples of practices and 
documentation 


| = Provide a means for organizations to measure themselves 
using a standard method (SCAMPI) 


= Provide a means to roughly compare achievements against 
other similar organizations (Maturity Levels) 


EMM) OrovViclss <) prilee. 
eli IIASELE: ERIE ae start ais ene telecine 
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What CMMI Does for NGMS? 


= Common model for understanding how to execute in complex 
development environments 


= Software and systems engineers now have an integrated model 
upon which to build an approach 


= Repeatable understanding of execution from project to project 
and from project managers to executives 


= Executive management has reference points to manage 
quantitatively and repeatedly 


= Measures are understood both by developers and reviewers 
= Estimates are based upon similar work and scope 
= Can easily compare similar programs 


= Personnel can migrate to the next project, even in a different 
division, and know what to expect and how they are expected to 
perform 


= Start up is easier and more predictable 


NORTHROP GRUMMAN 
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Wt What CMMI Doesn't Do! 
3 = CMMI (and Organizational Process) doesn’t mean that 


f, ; Sroieets will automatically use them 


= = New managers or those joining from other organizations 
> ) may not be convinced yet 


= CMMI doesn’t specify efficient and effective practices — it 
5 defines what to expect 
= Organizations have to create their own effective practices 


a 


= CMMI can’t guarantee error-free performance 


= Early detection of problems and effective responses are key 
to minimizing impacts 


= Even the best estimates based upon poorly understood 
requirements can’t be accurate. 


EMM releinels ars poe Silver oullses: 


Pe aa et 
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Project Management Maturity 


= Project Planning is where it all starts 
= Repeatable and executable to eliminate surprises 
= Estimating what it really will require 


= We focus on accuracy in estimation as a basis for 
determining the feasibility of achieving the desired 
results 


= Include factors to cover historical variations 


= Understanding your risks and how to mitigate them 
makes all the difference 


= Puts the project and the executives on the same page 


= Gives the customer a view of development risks that 
they normally can't get 


NORTHROP GRUMMAN 
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Striving for Repeatable Execution 


= Organizations carry techniques from one project to another 
— especially within core business areas 


= Organizational Training defines skill and process training 
for the organization and the project 


= Process adherence emphasizes repeatability 


= Organizational Process Focus minimizes unnecessary 
variants (when used with reason) 


= Project Planning is the means to create it 


= Quantitative Management and Project Monitoring and 
control are often used to ensure compliance 


= Enforcing use of organizational practices 
= Key to success from the outset 


= Start up kits provide the templates and direction starting 
with capture teams 


NORTHROP GRUMMAN 
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Building the right thing, the right way! 


= First you have to know what to build 
= Requirements Development & Requirements Management 
= Then you have to build it right 
= Technical Solution & Product Integration 
= And ensure it is built right and you built the right thing 
= Validation and Verification 
= But you fail if you don’t build it on time and within budget 
= Project Planning 
= Project Management & Control 
= Quantitative Management 


= And the initial estimate is made before the requirements are 
agreed upon (There is lots of room for underestimation) 


_USTOMEe CFUST GaINed VV ECE DeaAatTeCd SUCCeCSS - 
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Using “Inch-Pebble” Planning and Monitoring 


o, | Status on Sept 22nd | Status on Oct 6th | 
4 in | 3 


| % of Design 


<-- Complete 


<-- Complete 
' Suspended to atter re-design 


MIC Ate ri ale) In cloctei| pelos cat tno Sifore 
directed to where if celal do the Most ies & 


AA y ar 
Nav in Qa 1VP & GM Int sted agoesnt nurt eithne 
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Mitigation to Manage Schedule & Performance 


Status on Oct 6th 
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Results That | mpress Executives 


= During the last 4 years: 


= Projects started using NGMS mature processes haven't 
experienced pervasive problems 


= When & if problems do occur (and they will regardless of 
process) 


= Problems are identified before causing severe damage 


= Action is taken quickly to return the projects to effective 
performance 


= Requirements Development that uncovers 
misunderstandings is the biggest cause of variations 


= Since CMMI was adopted: 


= Customer funding problems are the #1 reason for appearing 
on the problem projects list 


= Most projects that are reported for performance problems 
predate CMMI organizational adoption 


ee ol oe ce Pe - ea pe yy 
QD | \ [my Nf | | : Nn ( 
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A Development Organization’s ROI in CMMI 


= An organization’s life blood is winning new projects 


= Estimation validity gains support from executive 
management to pursue aggressively and price 
competitively 


= Repeatable performance is key to support both when 
bidding and during development 


= Risk isn’t just a dirty word in Government 


= Executives view Proposal Risk, Program Risk, & 
Execution Risk equally 


= Identification and mitigation of risk using proven and 
trusted process Is key to gaining executive support 


= Individuals trained on mature programs migrate easily to 
new starts 


= Takes one more risk out of start up 


NORTHROP GRUMMAN 
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CMMI 
— Current Status 


— Version 1.2 was released on August 25, 2006. 


— Version 1.2 upgrade training is available 
online. 


— New policies announced and implemented for 
version 1.2 appraisals. 


— CMMI adoption continues to grow worldwide. 


4 * ® 
Cmmi 
Topics 


e Current Status 

¢ Appraisal Results 

¢ Transition Status 

¢ Product Suite V1.2 Improvements 
e Summary 
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Appraisal Synopsis as of 6/30/06 


¢ Based on SCAMPI v1.1 Class A appraisals 
conducted since April 2002 release through June 
2006 and reported to the SEI by July 2006. 
— 1,581 appraisals 
— 1,377 organizations 

840 participating companies 

— 109 reappraised organizations 

6,001 projects 

— 63.8% non-USA organizations 


¢ * Organizations previously appraised against CMMI v1.0 and who 
have not reappraised against v1.1 are not included in this report. 

e Please visit http://www.sei.cmu.edu/appraisal- 
program/profile/about.html, for additional information or to find 
answers to questions you may have about this briefing. 
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Wholesale Trade Transportation, 


Organizational Type 


Based on Primary Standard Industrial Classification (SIC) Code 


Retail Trade — 1.0% Communication, Electric, Gas 
0.4% L and Sanitary Senices 
Mining — \ ie 3.2% Finance, Insurance and Real 
0.2% : ; vs — Estate 
fo 5.9% 
/ 


Business Services 
40.8% 


Manufactu ring” 
23.8% 


Health Services 
Engineering & Management 0.6% 
Services | 
13.3% Other Services 


0.8% 


Based on 505 organizations reporting SIC code. For more information visit: 


http:/www.osha.govioshstats/sicser.html 


Services Pa 
55.4% ae 


Public Administration 
—— (Including Defense) 


10.1% 


Fabricated Metal Products 
0.4% 


_ Primary Metal Industries 
1.0% 


Industrial Machinery And 
— Equipment 
2.2% 


Electronic & Other Electric 
Equipment 
5.1% 


Instruments And Related 


ee Products 


5.9% 


Transportation Equipment 
9.1% 


x »® . - - 
cmmi Organizational Size 
Based on total employees within area of appraised 


1001 to 2000 
ae ~ 2000+ 
501 to 1000 | (2.8% 
poe : | / _ 25 or fewer 
\ \ / ae 9.8% 
201 to 2000+ 
37.0% 
301 to 500 1 to 100 - ee > 
9.5% 43.0% a 
, 51 to 75 
201to300 = re 
10.9% 101 to 200 
20.0% 
76 to 100 
8.7% 


Based on 1,348 organizations reporting size data 7 
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Reporting Organizational Types 


Commercial/In-house 67.6% 
Contractor for 
Military/Government 
Military/Government 
Agency 
0 200 400 600 800 1000 1200 


Number of Organizations 


Based on 1377 organizations 


Pe 
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CMMI ; oo ; 
— Disciplines Selected for Appraisals 


Number of Appraisals 


50.7% 
800 - 


MSS 


Based on 1,581 appraisals 


36.2% 


3.9% 


MS 
SSIMSIAS 


SS/dddlIMSAS 


Cddl/MS/AS 


SE = System Engineering 
SW_ = Software Engineering 


IPPD = Integrated Product and 
Process Development 
SS = Supplier Sourcing 


0.8% 0.2% 0.1% 0.1% 


SSIMS 
SSAS 
SS/dddl/AS 
SS/dddlIMS 


For more information about Allowable Models & Combinations, visit: http:/Awww.sei.cmu.edu/cmmi/background/aspec.html a 


Number Of Organizations 


450 


400 


33.3% 


33.8% 


Maturity Profile by All Reporting 
Organizations 


18.2% 


Not Given Initial Managed 


Based on most recent appraisal of 1,377 organizations 


Defined 


Quantitatively 
Managed 


Optimizing 
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CMMI! countries Where Appraisals Have Been 
Performed and Reported to the SEI 


—— ss oo : es 
Argentina Australia Austria Bahrain Belarus Belgium Brazil Canada 
Chile China Colombia Czech RepublicDenmark Dominican Republic Egypt Finland 
France Germany Hong Kong India Indonesia Ireland Israel Italy 
Japan Korea, Republic of Latvia Malaysia Mauritius Mexico Morocco Netherlands 
New Zealand Pakistan Philippines Portugal Russia Singapore Slovakia South Africa 
Spain Sweden Switzerland Taiwan Thailand Turkey Ukraine United Kingdom 


United States Vietnam 


Red country name: New additions with this reporting 11 


or Maturity Profile by All Reporting 
USA and Non-USA Organizations 


100% 


90% @ USA: 100 % = 498 


80% @ Non-USA: 100 % =879 


70% 


60% 


50% 


40% 


% of Organizations 


30% 


20% - 


10% 


0% 


Not Given Initial Managed Defined Quantitatively Optimizing 
Managed 


Based on 498 USA organizations and 879 Non-USA organizations 12 


CMMI Number of Appraisals Conducted by 


Year, as of 10/31/06 
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Appraisal Results Summary 


— 1,581 SCAMPI A appraisals have been reported to the SEI in 
52 months; 744 of these were reported in the last 12 months. 


— We believe that there are a significant number of non-SCAMPI 
appraisals being conducted. 


— Commercial/In-House, Government/Military, and 
Government/Military Contractor organizations reporting 
appraisals are increasing at a stable and consistent rate. 


— Comparing early reports of the SW-CMM maturity profile with 
early CMMI data reflects a more mature CMMI profile. 


— CMMI adoption continues to grow worldwide. 
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Topics 


¢ Current Status 

¢ Appraisal Results 

¢ Transition Status 

¢ Product Suite V1.2 Improvements 
e Summary 


ae 


cmmi _ 
— CMMI Transition Status, 10/31/06 
¢ Training 


— Introduction to CMMI —- 58,178 
— Intermediate CMMI - 2,280 


e Authorized 


— Introduction to CMMI v1.1 Instructors — 404 
— SCAMPI v1.1 Lead Appraisers — 451 


e Partners 


— Introduction to CMMI Training Services — 238 
— SCAMPI Appraisal Services — 259 
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Authorized (Cumulative), 10/31/06 
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CMMI Adoption Trends, Web Page 


VieWS, as of 9/30/06 


CMMI Web pages views in 
September 2006 


360K/month 


Exceeded 24K views on 27 
September 2006 


The following were the most 
downloaded files on the SEI 
Web site in September 2006: 


CMMI-DEV, V1.2 

CMMI V1.2 Overview 
Presentation 

“CMMI V1.2 Model Changes” 
Presentation 

“Extreme Programming (XP), 
Six Sigma, & CMMI: How 
They Can Work Together” 
Presentation 

Comparison of Version 1.2 
PAs to Version 1.1 PAs 


Average daily page views per quarter 


© © 
S SS 
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cmmi Performance Results Summary -1 
as of 8/30/06 


# of data 
Improvements points fm Low 


a 
seneduie | sm | 22 
Proauctviy | 63% | 20 


Customer 14% 7 A% | 55% 
Satisfaction 

pee A.0:1 1.7:1 |27.7:1 
Investment 


30 Organizations with results expressed as changes over time 


we 
CMMI 


— Performance Results Summary - 


¢ For more information on CMMI performance results, see 
— An August 2006 SEI technical report titled Performance Results 
of CMMI-Based Process Improvement (CMU/SEI-2006-TR-004) 


e It is available on the SEI Web site at 
http://www.sel.cmu.edu/publications/documents/06.reports/06tr004. 
html. 


— The CMMI Performance Results Web site at 
http://www.sei.cmu.edu/cmmi/results.html 
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— Topics 
¢ Current Status 
¢ Appraisal Results 


e Transition Status 


¢ Product Suite V1.2 Improvements 
— Identification of Issues 
— Resolution of Issues in V1.2 
— Summary of V1.2 Changes 
— Next Steps 


e Summary 
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renal Surfacing of Problems with 
Use of CMMI Ratings 


e Problems presented by OSD at the 2004 NDIA CMMI 
Technology Conference 


— When achieving a level replaces the focus on continuous 
improvement, we’ve lost sight of the goal 


— We created “level-mania” instead of continuous improvement 

¢ NDIA Workshop and Summit on CMMI Use in DoD 
Programs, Sept 2005 

— Explored issues, and identified a set of recommendations 
e Spring 2006 Government Review and Assessment 

— Validated Workshop findings 

— Refined and augmented issues, developed recommended 

actions 
— Reported to CMMI SG and Sponsors 


e Ongoing: DCMA data call on CMMI implementation. Initial 


assessment confirms problems 
22 
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ae Issue Areas 


e¢ Issue Area 1: High Maturity/Level 4&5 
— Lack of agreement on what constitutes Levels 4 and 5 
— High-maturity practices are not consistently applied at the 
project level after contract award 
e Issue Area 2: Integrity of Appraisal Results 
— Appraiser training, consistency, independence 


— Content of Appraisal Disclosure Statement (ADS) is not 
representative of CMMI appraisal data 


— Appraisal sampling inconsistent and influenced by appraised 
organization 
e Issue Area 3: Organizational Commitment 
— Concern that new projects are not incorporating standard 
processes 
e Issue Area 4: CMMI Guidance for Acquirers 


— Inadequate training and education for acquirers resulting in 


misrepresentation and misuse of CMMI _ 
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CmMmi 
— Topics 
¢ Current Status 
¢ Appraisal Results 


e Transition Status 


¢ Product Suite V1.2 Improvements 
— Identification of Issues 
— Resolution of Issues in V1.2 
— Summary of V1.2 Changes 
— Next Steps 


e Summary 
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Issue Area Resolutions -a 


¢ Issue Area 1: High Maturity/Level 4 & 5 


— Initial certification of high-maturity appraisers 


e¢ July 06 workshop on high maturity training determined the 
requirements lead appraisers have to have in order to do 
high maturity appraisals 


¢ Application review and oral exam 
— Body of Knowledge on leading appraisals will be 
available by October 2007 


— Lead Appraisers must certify that level 4 and 5 
appraised subprocesses map to organization’s business 
objectives 
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Issue Area Resolutions -b 


¢ Issue Area 2: Integrity of Appraisal Results 


— CMMI v1.2 upgrade training, proctored exam, and face- 
to-face training 

— Eliminated “level for life”’—now three-year limit 

— DCMA developed a survey related to CMMI appraisal 
ratings and program performance to begin to measure 
the problem 


— CMMI v1.2 updates to the ADS and to Sampling 
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Issue Area Resolutions -c 


¢ Issue Area 2: Integrity of Appraisal Results - 
continued 
— Precise definition of the appraisal sampling 
¢ Organizational scope: name, type, location 


¢ Organizational Unit Coverage: size, application domain, 
geographical breadth, project type expressed in 
percentages of total organizational unit 


e Projects excluded and rationale 
— Lead Appraiser certification that focus and non-focus 
projects are representative of organization 


— Lead Appraisers must come from an organization other 
than the business unit being appraised 


af 
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Issue Area Resolution -d 


¢ Issue Area 2: Integrity of Appraisal Results - 
continued 
— v1.2 ADS requires improved level of detail 
¢ Provides details on appraisal sample 
— Organizational unit, projects, domains 
Provides Lead Appraiser certifications 
— Project sampling is representative 
— Level 4/5 certifications are based upon practices that 
represent organizational business goals 
¢ All appraisals beginning after 1 November 2006 must use 
and fully populate ADS v1.2 (includes CMMI v1.1-based 
appraisals) 
¢ ADS for all DoD contractors will be posted (government 
Web site TBD) for Government acquirer review 
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Issue Area Resolutions -e 


¢ Issue Area 3: Organizational Commitment 

— Added new goal and two practices to Operational 
Process Focus (OPF) PA to stress deployment of 
processes to projects. 

— Added text in Integrated Project Management (IPM) PA 
to emphasize having a defined process at project start- 
up 

— CMMI Guidebook for Acquirers will discuss need to 
address project level implementation with developers 
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Issue Area Resolutions -f 


e Issue Area 4: CMMI Guidance for Acquirers 


— CMMI Guidebook for Acquirers 


¢ Will address how CMMI should be used by acquirers, how 
to interpret appraisal results, how to treat CMMI 
throughout the lifecycle 


¢ Due out after the first of the year 
— Development of the CMMI-Acquisition Constellation 
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— Topics 
¢ Current Status 
¢ Appraisal Results 


e Transition Status 


¢ Product Suite V1.2 Improvements 
— Identification of Issues 
— Resolution of Issues in V1.2 
— Summary of V1.2 Changes 
— Next Steps 


e Summary 
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— Reduced Model Complexity & Size 


¢ Eliminated the concepts of advanced practices 
and common features 


¢ Incorporated ISM into SAM; eliminated Supplier 
Sourcing (SS) addition 


¢ Consolidated and simplified the IPPD material; 
eliminated two process areas (OEI, IT) 


¢ Consolidated all definitions in the glossary 
¢ Adopted a single book approach 


e Reduced size 15% 


wi Increased Model Coverage 


¢ CMMI always covered hardware design, but the 
naming (CMMI-SE/SW/etc) didn’t reflect this 


e Added hardware amplifications 


¢ Added two work environment practices (i.e., one 
in OPD and one in IPM) 


¢e Added goal and two practices in OPF to 
emphasize importance of project startup 


e Updated notes on service development and 
acquisition of critical product elements 


¢ Updated name to CMMI for Development (CMMI- 


DEV) to reflect the expanded coverage 


a 
ape Appraisal Method Changes to 


Improve Integrity 


¢ Method implementation clarifications 
— interviews in “virtual” organizations 
— practice characterization rules 
— organizational unit sampling options 
¢ Appraisal Disclosure Statement (ADS) 
improvements 
— improve usability for sponsor and government 
require sponsor’s signature on the ADS 


require all team members to show agreement on 
findings via signature 


— Both v1.1 and v1.2 ADS reflect these changes now 


¢ Appraisal team will have responsibility for 
determination of “applicability” for SAM; no other 
process area exceptions will be accepted for 
maturity level achievement 34 


nny Appraiser Qualification Changes to 
Improve Integrity 


¢ All v1.2 Lead Appraisers must do the following: 
— pass upgrade and refresher testing 
— attend “face-to-face” sessions on appraisal quality 
— pass an additional oral exam before conducting a high 
maturity (HiMat) appraisal 
¢ All v1.2 Appraisal Team Members must 
accomplish upgrade training and be registered at 
SEI 


¢ All v1.2 high maturity appraisals must be led by a 
certified high maturity Lead Appraiser 


i 
CMMI 


CMMI Training v1.2 


¢ Introduction to CMMI 

— Updated for v1.2 

— Phased in this fall 
e Intermediate Concepts of CMMI 

— Updated for v1.2 

— Will better prepare students for SCAMPI training 
e CMMI Instructor training 

— Updated to reflect v1.2 changes 


“Delta” training from v1.1 to 1.2 
— Available online for free 
— More extensive upgrade course for fee 
¢ Appraisal Team Members 
e Lead Appraisers, instructors, candidates 
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e Transition Status 


¢ Product Suite V1.2 Improvements 
— Identification of Issues 
— Resolution of Issues in V1.2 
— Summary of V1.2 Changes 
— Next Steps 


e Summary 
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CMMI V1.2 Schedule 


Version 1.2 CMMI Product Suite 
release 


Update material available 
Upgrade course available on-line 
First Lead Appraiser “face to face” 


V1.2 ADS required for all SCAMPIs 
(on-site starts) 


Last v1.1 Intro training 


First expiration of v1.1 appraisals (3 
year validity) 


Last v1.1 appraisal (on-site starts) 


August 25, 2006 
August 25, 2006 
August 25, 2006 
October 16, 2006 
November 1, 2006 
December 31, 2006 
August 25, 2007 


August 31, 2007, 


i 
CMMI 
os CMMI “v2.0” 


¢ Sponsors and CMMI SG beginning discussions 
on CMMI v2.0 


e Plan on series of focused NDIA/SEI Workshops in 
2007 to dialog strategies on both models and 
appraisal methods 

— Need to see effects of v1.2 changes 


— Need to see how we can improve efficiency and reduce 
CMMI adoption costs 


— Need to determine if any better methods can be 
implemented 


— Need to continue the analysis of CMMI integrity 
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~4 Summary 


¢ CMMI has clearly reached far more users than 
ever envisioned. 
— We need to be sensitive to this as we move forward 


¢ CMMI Adopters are reporting increased 
performance. 


¢ CMMI v1.2 has made substantial improvements to 
address integrity issues: 
— Model 
— Appraisal Method 
— Appraisal Reporting 
— We will continue to monitor the integrity-driven 
improvements for effectiveness, and react accordingly 


¢ We welcome your feedback. P 
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Practical Experiences and Lessons 
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Director Engineering Operations Code 09K SEI Authorized CMMI Trainer 
Department Head Code 70 SEI Authorized SCAMPI Appraiser 
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ee Presentation Outline 


Systems Center 
Charleston 


> Introduction to SPAWAR Systems Center Charleston 
> Vision and Strategy 

> Critical Success Factors 

> Practical Experiences 

> Success! 

> Lessons Learned 

> Going Forward 

> Summary 
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VE. Introduction to SSC-Charleston 


>Where we fit 
>What we do 
>What we are known for 


>Who we are 
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VE Where We Fit 


Systems Center President 


Charleston 
——— — Biploysy Ble) D) 


Secretary of Defense 


SPAWAR 


Space and Naval Warfare 


Systems Command Secretary of the Navy gummi Other DoD 
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Fleet Support PNexelUiiitel a 


NETWARCON] MARCOR ) SPAWAR NAVSEA NAVAIR NAVSUP NAVFAC 
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SYSCENY, 
Charleston, SC 


Network Centric 
Enterprise 


Swe 


SPAWAR What We Do 


4 


Systems Center 
Charleston 


Ove) alatcreqdlale Meatemulcclaile laid 


Mission- We enable knowledge superiority to en Ge <a) | 
Naval and Joint Warfighters through the a: a <= haraeeiae 
development, acquisition, and life cycle support] [Ay Se eaten “RE Technology 
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capabilities. 
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VE. What We’re Known For 


Systems Center 
Charleston 


«Developer of FORCEnet joint collaborative 
assessment tools that promote netCentric 
interoperability and reduce system redundancy 


¢Principal SPAWAR provider for Joint and 
Homeland Security C41 solutions in a responsive 
manner 


S 
2 
Ss 
o 
= 
oo 
— 
= 
° 
@ 
a 
= 
4 
L 
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eNavy’s most efficient provider of critical 
engineering and acquisition expertise for Navy/Joint (gaseous 
commands and other federal agencies 


eRapid integrator and deployer of interoperable technologies to the 
Navy, Federal Government, and Joint Warfighter 


¢eDeveloper and employer of life-cycle logistic support solutions ina 
web-enabled portal environment 


SEI 
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VE Who We Are 


Systems Center 


Charleston A Large Systems & Software Engineering Organization 


Computer 
Science/ 


=tatetiatcr-dalale p 


Over 70% of 
workforce is in an 


Science & 
Engineering, 


1049 


engineering or 
computer-related 
discipline 


wyye—Contracts & Supply, 112 


©————__ Other, 174 


Program Mgmt, 106 Logistics, 79 


¢ The solutions to the global war on terror developed by SPAWAR 
result from good systems and software engineering. 


¢e Systems engineering is our core competency. 
¢ Total workforce of ~ 2,300 employees. 
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VE. Vision 


Systems Center 
Charleston 


¢ Vision 
— Develop and Maintain a World Class Systems Engineering 
Organization 


¢ Strategy for Implementing CMMI® 


— Research Best Methods (Case Studies) 
— Investigate Techniques and Models 

— Build Plan of Action 

— Implement Plan of Action 


=> SEIPartner 
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VE Research and Investigate 


Systems Center 
Charleston 


— Extensively researched industry and government 
organizations that have successfully used the 
CMM® and CMMI® to implement process 
improvement* 


— Identified commonality among implementation 
approaches and lessons learned 


— Identified Benefits of CMMI® 
— Identified “Critical Success Factors” 


* Case Studies Included: Boeing-Integrated Defense Systems (IDS); U.S. Army Armaments Research, 
Development and Engineering (RDE) Centers; Lockheed Martin Corporation; Electronic Data Systems 
(EDS); Raytheon; Northrop Grumman - Mission Systems; Motorola - Global Software Group; General 
Dynamic Advanced Information Systems; SPAWAR Systems Center San Diego; Defense Finance and 
Accounting Service; Thales Training and Simulation; Jet Propulsion Laboratory; Bosch Automotive; 
Jacobs Sverdrup 


SEI 


WecHsort 


Technical Software Services, Inc Approved for release to the public - 12 Oct 2006 9 


VE Benefits of Implementing CMMI®! 


Systems Center 
Charleston 


¢ Increased control of requirements, costs and schedule 


¢ Increased ability to predict schedule and costs of product 
and product components 


¢ Ability to remove defects early and efficiently from the 
work products 


¢ Reduced rework leading to reduced development cycle 
time 


¢ Increased predictability and control of Bek quality (the 
uality of a system Is highly influenced by the quality of 
the process used to acquire, develop, and maintain it) 


WecHsort 
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VE Benefits of Implementing CMMI® 


’Gharieston 
e Enhanced ability to make cost-benefit trade-offs of 
implementing new technologies and processes 


° Melee Capability to select and manage qualified 
suppliers 


¢ Enhanced ability to make management decisions based 
on quantitative data 


¢ More time available for top innovators to spend on 
problems and challenges requiring creative energy 


e Enhanced communication and involvement of everyone in 
continuing process improvement efforts 


WecHsort 
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VE CMMI® 


Systems Center 
Charleston 


() 
SSC-Charleston chose to Sgod Replosontaton 
implement CMMI® because ae 


Hp 
It provides a structured | | Orvntationt are colon and Deslal 
model for process Manage) 


improvement and is used Defined (2) Dein Avalian Realuton 
to measure an d | mM p rove an Integrated Project arg 


Organizational Trai 


O rg an izZatl on is abl | ity to ra erp: Process Focus 


Verification 
successfully manage Peet neato 
Requirements Development 


C O mM p | ex syste mM S Organizational Process Definition 


engineering and software omnis Configuration Management 


4 Process and Product Quality Assurance 
if O e cts Supplier Agreement Management 
p | . Project Monitoring and Control 
Project Plannin 


Requirements Managers 
Measurement and Analysis 


The model recognizes ve Benet i ae and ha i 
e xC e| | e n C e | n b U S| n e S S competence and heroics but is not consistent. 
practices, aS measured ae 
against a set of demanding or 


criteria. 
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VE. Critical Success Factors 


Systems Center 
Charleston 


CRITICAL SUCCESS FACTORS FOR IMPLEMENTING CMMI® 


Commanad-wide Policy Assign Responsibilities 
(Create vision that is urgent) (Strong Change Agents are essential) 


Strategy and Plan (Include 
knowledge of why change is 
necessary and benefits) 


Provide Training 


Senior Management Support Build Central Repository 


Provide Resources and Funding 
(New Organizational Structure Measure and Communicate Progress 
Usually Needed) 


Wecusort 
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W, Practical Experiences 


Systems Center 
Charleston 


Applied the Critical Success Factors: 
1. Ensure Policy Published at Highest Level 
2. Obtain Approval for Process Improvement (PI) Plan 


3. Obtain Resources (Funding) and Assign Responsibility for PI 
Initiative 


Build Support for the PI Initiative 
Plan and Provide Training 


Build and Maintain Central Repository 


a + 2S 


Measure and Communicate Progress 
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‘WecHsort | 
Technical Software Services, Inc. Approved for release to the public - 12 Oct 2006 14 


VE. 1. Policy Published at Highest Level 


Systems Center 
Charleston 


Si Command-wide policy signed by our Technical 
Vf Director, approved by the Board of Directors, and 
published December 2003. 


— The policy directs the use of the best practices represented 
in the CMMI®-SE/SW model for all SSC-C systems and 
software engineering projects and tasks. 


— The policy also directs the use of industry standards 
(ISO/IEC 15288 for Systems Engineering and ISO/IEC 
12207 for Software Engineering). 


WecHsort 
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2. Get Plan Approved 


J > Process Improvement Plan and Schedule 
MI Approved February 2004. 
— Plan included why changes were necessary 


— Schedule included achievement of CMMI® Maturity Level 2 
for Command in April 2005 


— Schedule includes achievement of CMMI® Maturity Level 3 
for Command in April 2007 
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W, Practical Experiences 


Systems Center 
Charleston 


Applied the Critical Success Factors: 
1. Ensure Policy Published at Highest Level 
2. Obtain Approval for Process Improvement (PI) Plan 


3. Obtain Resources (Funding) and Assign Responsibility for P| 
Initiative 


Build Support for the PI Initiative 
Plan and Provide Training 


Build and Maintain Central Repository 


~~ & FS 


Measure and Communicate Progress 
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VE 3. Resources and Responsibility 


Systems Center 
Charleston 


J > New Organizational Structure Established and 
Vf Funded at the Command Level. 
— Director of Engineering Operations (Code O9k) 


— Engineering Process Office (EPO) 


— Command and Departmental Engineering Process Groups 
(EPGs) 


— Various Integrated Process Teams (IPTs) 
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New Organization for Implementation 


‘E 


Systems Center 
Charleston 


M. Kutch 
Dir. of 
Engineering 
Operations 


apiece 


Process Office )—~ 


Enterprise 
Process Group 
(Ent PG) 
Codes 09K | 09A 
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W, Practical Experiences 


Systems Center 
"Charleston 
Applied the Critical Success Factors: 
1. Ensure Policy Published at Highest Level 
2. Obtain Approval for Process Improvement (PI) Plan 


3. Obtain Resources (Funding) and Assign Responsibility for Pl 
Initiative 


Build Support for the PI Initiative 
Plan and Provide Training 


Build and Maintain Central Repository 


a 2 2 oS 


Measure and Communicate Progress 


SEIF 
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Y, 4. Build Support for PI Initiative 


Systems Center 
Charleston 


LM Spread the word! 


— Shared Early Successes in The Chronicle, SSC-C’s site 
publication 


— Built Senior Management Support 


— Created a Newsletter Focusing on Systems & Software 
Engineering Process Improvement 


e Available in printed and electronic format 
e Published every 2-3 months 
— Provided Extensive Mentoring and Coaching 


— Communicate what is expected from SSC-C Projects 
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Spread the Word! 


systema contr SyStems and Software Engineering Newsletter 


Charleston 


Systems and Software Engineering Newsletter 
Volume 2, Inver 1 February 2006 


The Benefits of CMMI® 


SSC Charleston's Project Managers have pushed their tears to practice CMNMJ® and are 
reaping great benefits as a result. The Engineering Process Office sat down with thern to 
discuss what they learned during their pursuit of Maturity Level 2. Article on Page 2. 
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KUTCH’S KORNER: CMMI” Makes SSC-C Work Smarter 


be thee pet you, we've token great miepe toward meking SPAWAR Systeens Coster 
Charleacn 4 world-class exguntering cegeniation We had ax towns go to CMO4I* 
Mataity Level 2, and then expenenoe 9 Qe focus of thie waue of the ¢ TNT 

On anted, dis effcet required ugedficent work, but SSC4 wo scemg gumercu benefits 
ee crete of nplemmenting ONO By jumping adeed Uerse towns have peowided « 
toudenep we all cas Follow, allewsng us to work smarter to acheove the seme succem 


The tangsbhe, aaentfiade benefity we we teaged by umnplementing CMO have 
tensed SSC Charbestan s regratatson — beth wm the Command and Ge crvkan work! - 
as 6 qualey exgimerring trettution end Gat's sencibing we cen all be proud of Te 
thets inter veews Bor the astche you're about to read, SECs propeets were very 


= => sEIPartner 
‘Wecusort 


Technical Software Services, Inc. Approved for release to the public - 12 Oct 2006 22 


W, Practical Experiences 


Systems Center 
"Chaviaeion 


Applied the Critical Success Factors: 
1. Ensure Policy Published at Highest Level 
2. Obtain Approval for Process Improvement (PI) Plan 


3. Obtain Resources (Funding) and Assign Responsibility for Pl 
Initiative 


Build Support for the PI Initiative 
Plan and Provide Training 


Build and Maintain Central Repository 


a + 2 


Measure and Communicate Progress 
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ee 5. Plan and Provide Training 


Systems Center 
Charleston 


= 

LVF eIntro to Process Improvement 

ML —Formal Classroom Course Initially 
— Currently Provided via WBT 


— Now Mandatory for all employees 


eCMMI® 
— SEI Intro to CMMI® 
—SSC-C Level 2 Processes 


¢eSystems Engineering Fundamentals 
Classes 
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Systems Center 
Charleston 


SPAWAR 
WA introduction to SSC-C Process Improvement 


eTraining 


O Courseware Operations 

O Course Introduction 

C1 Introduction to Process Improvement 

O Terminology ‘ainally qi di 

Gar ere Originally given as a podium 
get eee ee course, converted to Web 

ED SSee Pe sn Based Training in 2004 

O Organizational Implementation N ee tor all 

O Level 2 Process Manuals ON Te QUIe 10a 
eee ee employees 

O Course Summary 
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VE SEIl’s Intro to CMMI® 


Systems Center 
Charleston 
e SEI’s Introduction to CMMI® course teaches the 
full CMMI® model 


— Students learn how the best practices build and 
relate across process areas 


— Learn the terminology and complete model 


— SEl-Authorized instructors are well-versed in our 
implementation to augment material with SSC-C 
specific content 


— Highlight SSC-C tools and resources 


¢ Over 300 employees trained 
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SPAWAR Systems Engineering 
(A Fundamentals Classes 


Systems Center 
Charleston 


¢ Teach the Systems Engineering process 


¢ 3-day on-site, classroom course : ¥ 


— Based on SMU SE Masters course 
— Customized to incorporate SSC-C SE process 
— Over 300 SSC-C engineers trained 


e 1-day SE for Managers course added 


“Thought provoking, motivating, and challenging. Learning basic SE 
caused me to brainstorm many different applications of organized 
system processes. It motivated me to want to begin organizing its 
application. It also challenged me to apply GOOD SE practices in order 
to successfully be more efficient in the process..” 


“It was extremely beneficial to have a professor with extensive 


knowledge of the subject matter and one who could apply it to the 
SPAWAR methods.” 


Student Feedback 


Wecusort 
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W, Practical Experiences 


Charleston 
Applied the Critical Success Factors: 
1. Ensure Policy Published at Highest Level 
2. Obtain Approval for Process Improvement (PI) Plan 


3. Obtain Resources (Funding) and Assign Responsibility for Pl 
Initiative 


Build Support for the PI Initiative 
Plan and Provide Training 


Build and Maintain Central Repository 


-— 2 eS 


Measure and Communicate Progress 
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VE 6. Central Repository 


Systems Center 
Charleston 


Li7 Built and Continue to Populate Central 
MI Repository (CORPWEB/CMMI® intranet website). 
— Policies & Process Manuals 
— Standard Operating Procedures (SOPs) 
— Sample Documentation and Templates 
— Projects’ Artifacts 
— Artifacts from Teams — IPTs, EPGs 
— Link to PI-WBT and other WBT courses 
— Links to Reference materials and guidebooks 


— Link to ePlan Builder (EBP) - Our CMMI Tool 
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Systems Center 
Charleston 


Engineering Process Office Website 


| Address [2 https: }/corpweb2,spawar.navy.mil/cmmi/standardprocesses/standardprocesses. aspx | Go | 


SPAWAR 


Systems Center 
Charleston 


Navigation 


Calendar 


SSC-C Standard 
Processes 


Systems 
Engineering 
Process 
Software 
Engimeering 
Process 

life Cycles 


Tailoring Criteria 


Process Areas 
Projects 
Process 
Improvement 
Teams 
Organizational 
Measurement 
Repository 
Training 


Innovation 
Pragram 


SSC-Charleston Engineering Process Office 


EPO Home | ePlan Builder | WBT Courses | eWBS | Contact EPO | CorpWeb 


SSC-C Standard Processes 


Currently, the SSC-C Standard Processes contain policies, process manuals for CMMI®-SE/SW Level 2 
and Level 3 process areas, and select SOPs. The standard processes for Systems Engineering and 
Software Engineering provide SPAWAR - E 

detailed procedures for ~ y, SSC-C Engineering Process Guidance 
accomplishing tasks within ee 
these respective disciplines. "=<" SPAWAR Instructic eer SPAWAR Policy 

The 3 top-level standard = oo: 
engineering processes are: 


sancras fepiess ih 


Policies mandate the use of ISO/IEC life cycle processes 


e Systems Engineering and CMMI wan” est pinotionst 
e Software Develooment Pt = 
e Software Maintenance 

These processes were Na Cees tt SE 


derived from the ISO/IEC 
industry standards to address 
the typical engineering work 


SSC-C Software 
Engineering Process 


Top Level 
SSC-C Processes 


sec-c ——— 
Engineering Process 


Ari 0 SE/SW Proce: 


performed by ssc-c, ,  REGM | PPOA MA 
Additional SSC-C standard IPC T 
processes have been e Val 


developed to further refine 
these top level engineering 


processes and to support the process areas of CMMI®, The graphic depicts the derivation of the SSC-C 
standard processes, 


a 
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Related Links 


Process 


Improvement 
Communication 
Plan 


SSC-C Artifact 
Submittal Form 


a 
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W, Policies and Processes 


Systems Center 
Charleston 


PNAS Tal ie [6 (-meo ys) Ors Ov al-lal-t-) co) alm ade) |(eavar-lale mm aa gelerct-t-m \UEclalerel 


addresses each of these Process Areas: 


e Project Planning ¢ Product Integration 
e Project Monitoring and ¢ Verification 
Control ¢ Validation 


* Configuration Management —«. Risk Management 
¢ Process and Product Quality . Decision Analysis and 


Assurance Resolution Management 
* Measurement & Analysis * Organizational Process Focus 
* Requirements Management __. Organizational Process 
e Supplier Agreement Definition 

Management ¢ Organizational Training 


* Requirements Development __. |ntegrated Project Management 
¢ Technical Solution 
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W, Practical Experiences 


Systems Center 
Charleston 


Applied the Critical Success Factors: 
1. Ensure Policy Published at Highest Level 
2. Obtain Approval for Process Improvement (PI) Plan 


3. Obtain Resources (Funding) and Assign Responsibility for Pl 
Initiative 


Build Support for the PI Initiative 
Plan and Provide Training 


Build and Maintain Central Repository 


a. 2 oY 


Measure and Communicate Progress 
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a7 4 * 7, Measure and Communicate Progress* 


Yétarieston 
SPF Progress Measured Every 1- 4 Months 


— Projects conducted Process Reviews and 
Document Reviews to measure progress 
and identify gaps using Apprasial tool 


—EPO performed Class B/C appraisals of 
selected projects 


— SEI performed Standard CMMI® Appraisal 
Method for Process Improvement 
(SCAMPIS") Class A’s at the Project-level 


— SEI performed Command-wide SCAMPIS™ 
Class A appraisal in April, 2005 
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a7 4 7. Measure and Communicate Progress? 


Systems Center 
Charleston 


LM Recognize and Publicize Early Successes 


—‘Project-level’ SCAMPIs provided early 
successes due to conducting the appraisal using 
the “continuous representation” of the model 


¢ Scope of appraisal looked at all 7 ML2 PAs and if the 
PAs were satisfied, I.e., achieved CL2, then the project 
achieved ML2 [equivalent staging] 


¢ Projects received CL2 for various PAs (e.g., CM, SAM, 
REQM, PP, PMC) 


—Led to BIG success! - SSC-C became the first 
SPAWAR Systems Center to achieve CMMI® 
Maturity Level 2 (April 2005) 
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VE Capability verses Maturity Level 


“weer” ~~ Continuous _ Staged 
... Rating is for a single 
process area ... Rating is for a specified 


set of PrOCESS areas 


ML5 


ML4 


ML3 


ML2 


Process Area Capability 
CLO Cli Cl2 be CL4 Crs 


PA PA’ PA 
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SPAWAR 
Vv, 


Systems Center 
Charleston 


WecHsort 


Success! 


First SPAWAR Systems Center to Achieve 
Command Level CMMI® Maturity Level 2 


World Class 
Systems 
Engineering 


Systems 
Center 
Charleston 


Capability 
Maturity 
Model 
Integration 
(CMMI®) 


MATURITY LEVEL 
April 28, 2005 


Also, First SPAWAR Systems 
Center to have a Program Achieve 
CMMI® Maturity Level 3 
(July 2006) 
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NG Question? 


Systems Center 
Charleston 


What do these Critical Success 
Factors and the model itself 
have in common? 
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VE Answer! 


Systems Center 
Charleston 


“Both Institutionalize the PI/CMMI® Process” 


Critical Success Factors |CMMI® Generic Practices 


Ensure Policy Published at Highest Level 2.1 Establish an Organizational Policy 
Get the Plan Approved 2.2 Plan the Process 


Get Resources (Funding) and Assign 2.3 Provide Resources 
Responsibility 

Get Resources (Funding) and Assign 2.4 Assign Responsibility 
Responsibility 


Responsibility Stakeholders 


Get Resources (Funding), Assign 2.10 Review Status with Higher Level 
Responsibility and Communicate Progress Management 
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VE Lessons Learned?! 


Systems Center 
Charleston 


1. Do your homework: Researching what others 
have done to successfully implement process 
improvement and what challenges they 
encountered helped prepare us. 


2. Formulate a good plan: Building a Plan based on 
the “Critical Success Factors” led to our success. 


3. Policy needs to be Top-down: Having Command- 
level policy energized the PI initiative. 


4, Train, train, train: Providing an understanding of 
what the CMMI is all about, what SE is all about, 
and how to implement within a project is critical. 
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VE. Lessons Learned? 


Systems Center 
Charleston 


5. Train some more: Train process owners (PPQA, 
CM, REQM, etc.) to be subject matter experts 
(SMEs). 


6. Bite off small pieces: Approach change in small 
steps and use those experiences, successes and 
‘best practice’ artifacts to ease the change for all 
Command personnel. 


7. Ensure they know what’s in it for them: Value 
added must be visible. Share benefits that others 
have experienced in implementing CMMI®. 


8. Advertise successes early on: Publicize each 
positive outcome as progress is measured. 
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VE Lessons Learned? 


Systems Center 
Charleston 


9. Full-time SME to mentor and coach: Dedicated 
Engineering Process Office (EPO) maintained 
momentum and resolved issues. 


10. Build a support system: Identify key “change 
agents” within the organization to overcome 
resistance to change (those most respected and 
energized). 


11. Make sure the plan gets implemented: 
Promulgate realistic timeframes to all stakeholders 
(EPGs, IPTS, projects) and monitor schedules for 
continued successes in the PI Program. 
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VE Lessons Learned‘ 


Systems Center 
Charleston 


12. Establish organizational assets early: 
Developing Process Manuals, Naming 
Conventions, Formats/Style Guides, Templates 
and Tools provided value, consistency and ‘starting 
points’ for projects. 


13. Communication is constantly needed: Use 
multiple methods and channels for effective 
understanding, up to date status, and cross- 
communication among teams. 
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i 


Systems Center 
Charleston 


The rest of the story......... 


Phase 2 - ML2 to ML3 


WecHsort 


VE. The Second Wave — ML2 to ML3* 


Charleston 
e Shift Focus of Process Improvement Strategy from 
“Implementing CMMI®” to “Executing Sound Systems 
Engineering” 


¢e Educate Project Managers on What's Expected 


¢ Improve Project’s Planning and Documented 
Processes 


¢ Provide additional CMMI® Training and WBTs 


¢ Incorporate ISO, Lean Six Sigma (LSS) and Balanced 
Scorecard (BSC) Initiatives 
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/ The Second Wave - ML2 to ML3? 


Charleston 
¢ Develop internal “self-assessment” process for measuring 
ongoing implementation of Maturity Level 2 processes 
¢ Populate EPO/CMMI® Website with ‘best examples’ 
¢ Implement Tailoring Guidelines 
¢ Implement SSC-C Measurement Repository 
¢ Implement ML2-to-ML3 Action Plans 
¢ Continue to Measure and Communicate Progress 
e Maintain Momentum and Commitment to Goals 
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SPAWAR SSC-C SE Revitalization Plan 


w 


Systems Center 
Charleston 


a (sYo0 l=] a1 ecme) mete) Or Ome) om acc\Viitclipsclilela 


Training / Education 
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SSC-C SE Instruction 


SSC-C SE 
Process Manual 


SSC-C SW-Dev 
Process Manual 


SSC-C SW-Maint 
Process Manual 


Intro to Pl WBT CMMI® Level 2 


E 101 WBT 
eB CMMI® Level 3 


SE Fundamentals 
Balanced Scorecard 


SE for Managers 


Project & Process Lean Six Sigma 


EPO Website Workshop 


Integrated Product 


Intro to Software Engr. eams 


ePlan Builder 


Architecture Dev. WBT IT Tools 


" 


Underway 
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‘E 


Systems Center 
Charleston 


SSC-C Engineering Process Guidance 


sas 


To be replaced with 
SSC-C SE/SW Instruction 


Policies mandate the use of ISO/IEC life cycle processes 
and CMMI best practices 


ISO/IEC 15288 ISOMEC 12207 


System Life Cycle Software Life Cycle 
Processes Processes 


CMMI® for SE/SW 


SSC-C Systems SSC-C Software T 
; op Level 
E P E P 
ngineering Process ngineering Process SSC-C Processes 


CMMI® Supporting Process Areas to SE/SW Processes 


v2 Sse CMMI° to 


Measure & Assess 


| OPF | oPD | oT | IPM | Processes 
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Classic System Engineering “Vee” Diagram 
Aligning SE with CMMI and Process Improvement 


Systems Center 
Charleston 


CMMI Process Areas 
etter align with SE “Vee” Demonstrate and Validate 
qu Pp System to User Validation 


Validation Plan 


Perform System Verification 
to Performance 
Specifications 


Develop System 
Performance Specification 
And System Validation Plan 


Expand Performance 
Specifications into Cl 
“Design-to” Specifications 
and Cl Verification Plan 


Assemble Cls and perform 
Cl Verification to Cl 
“Design-to” Specifications 


Evolve “Design-to” 
Specifications into “Build-to” Inspect to “Build-to” 
Documentation and Documentation 

Inspection Plan 


Fabricate, Assemble, and Code to “Build-to” 
Documentation 


time 


Design Engineering 


———_—_-> 
PP PMC = CM REQM -PPQA MA SAMs RSKM_—DAR © 
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VE. The Second Wave — ML2 to ML3 


Veneteston 
e Shift Focus of Process Improvement Strategy from 
“Implementing CMMI®” to “Executing Sound Systems 
Engineering” 


e Educate Project Managers on What’s Expected 


¢ Improve Project’s Planning and Documented 
Processes 


¢ Provide additional CMMI® Training and WBTs 


¢ Incorporate ISO, Lean Six Sigma (LSS) and Balanced 
Scorecard (BSC) Initiatives 


; = setPanner 
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VE What is expected from SSC-C projects? 


Systems Center 
Charleston 


«Execute sound systems engineering 
— Utilize the organizational processes, tools, guidance available 


¢ Sound systems engineering requires 
— Proper Project Planning 
— Control over the project and process 
¢ Configuration Management, Requirements Management 
¢ Verification (Peer Reviews), Process Quality Assurance, ... 
— Iterative and robust design and development methodology 
¢ Decomposition of Requirements, Analysis, Alternatives, ... 


— Frequent interaction of senior management and customers 
(Sponsor) 


CMMI® provides a standard to assess progress against 


sound systems engineering practices 


Wecusort 


Technical Software Services, Inc. Approved for release to the public - 12 Oct 2006 50 


Systems Center 
Charleston 


Gather Originating 


* Initial Capabilities 
Document Oo or 
Statement of 


Requirements 


ork (SOW) 


Validate Project Scope 
¢ Size; Project Criteria 


Invoke Appropriate 
Engineering Process(es) 
(Soa 


DoD 
>I 
= c 
o 
54 8 
Do a 
£O 0 
Ww 6 Oo 
Neo 
EW 
o 
~ Yn 
a 8 
no WH 


* ePlan Builder can be used to 
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See Soft. Dev. Proc. Manu 


generate this type of plan. 


SSC-C Project Process - Planning 


Review/Apply Tailoring Guidelines 
¢ Tailoring Form; Waivers 


Develop 
Project Management Plan (PMP) * 


(PP/PMC/IPM) 


Develop Project Management Support Plans 


* REQM Plan * ¢ SAM Plan 

¢ CM Plan * ¢ RSKM Plan 

¢ M&A Plan * ¢ DAR Plan Components 
* PPQA Plan * of the 


Taltcxele-iiaze| 
Project 
Develop Engineering Plans (as required) Management 


Package 


Systems Engineering Mgmt Plan (SEMP) and/or (IPMP) 


Systems Engineering Plan (SEP) * and/or 
Software Development Plan (SDP) 


(Address RD/TS/PI/VER/VAL) 


Develop Any Additional Required Plans 
Detailed VER/VAL Plan; Install Plans 
(BESEP); TEMP/SOVT Plan; Cut-Over Plan 

Other plans, etc. 


a 
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v, Pitfalls in Developing Plans 


Systems Center 
Charleston 


¢ Generating from scratch 
— Author specific 
— What did | forget? 
— Time — “Re-inventing the wheel” 
¢« Copy from “good” example 
— Is it good? 
— Cut and Paste errors — old project data 
— What did they forget or intentionally leave out? 
— Is my project really the same? 
¢ Document Outlines/Templates 
— Provide placeholders; not explanation/examples 
— No validation 
¢ General issues with all above alternatives 
— Consistency from Plan to Plan 
— Formatting 
— What belongs in this section? 


SEI 
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VE ePlan Builder Tool 


v, a-ak  ePlan Builder © 
id Systems Center 


Charleston 


Electronic CMMI® Compliant Documentation Application 


~ Sponsored b by the Director of Engineering Operati ions (O9K) - Mi chael Kutch 


ePlan Builder tool 


— An interactive, web-based application that leads the user 
through a structured interview process (like TurboTax®) 
to generate a CMMI®-compliant plan 


— Includes standard, consistent text 
¢ Definitions 
e Acronyms 
¢ Roles and Responsibilities 
— Generates an initial project-specific document 


SEIF 
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VE TurboTax® Usability 


Charleston 
¢ TurboTax® (and other income tax software) 


— Simplifies the process for completing income tax 
returns 


— Uses an “interview” format with the user to collect 
information 


— Provides common defaults (that can be changed) 


— Branches to necessary sections based on input 
collected; skips unnecessary sections 


— Information entered once, but used in several places 
— Validates information for gaps and inconsistencies 

— Provides page/field specific help text 

— Automatically generates final forms 
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VE Apply Concept to Building Plans 


Charleston 
¢ Collect common project information once 
— Ensures consistency from plan to plan 
¢ Acts as on-line template 
— Ask questions, give choices, prompt for input 


— Application logic ensures full coverage of required 
sections (from standard process manuals) 


— Logic permits skipping / branching as appropriate 
— Provide on-line, section sensitive help 

¢ Standard content can be included in all plans 

¢ Automatically generate initial draft of plan 
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VE ePlan Builder Tool? 


Systems Center 
Charleston 
ava arfare 
a’ 4 Systems Center ePlan Builder * 
Charleston Electronic CMMI® Compliant Documentation Application 


‘ Sponsored by the Director of Engineering Operations (O9K) - Michael Kutch 


¢ Generates CMMI®-compliant (Maturity Level 3) Plans: 
¢ Project Management Plan (with WBS) 
¢ Configuration Management Plan 
¢ Quality Assurance Plan 
¢ Requirements Management Plan 
e Systems Engineering Plan (DoD SEP Format) 
¢e Measurement and Analysis Plan 
e Systems Engineering Management Plan (under development) 
¢ Verification Plan (under development) 
¢ Decision Analysis and Resolution Plan (under development) 


SEtPartner 


= 
‘Wecusort 


Technical Software Services, Inc. Approved for release to the public - 12 Oct 2006 56 


VE ePlan Builder Tool? 


Systems Center 
Charleston 
VY oe core ePlan Builder © 
Charleston Electronic CMMI® Compliant Documentation Application 


* Sponsored by the Director of Engineering Operations (O9K) - Michael Kutch 


¢ Generates plan in Rich Text Format (rtf) 
— For further editing and updating 


¢ Generates SSC-C compliant plan-specific 
standard lists for: 


— Definitions 


_ Acronyms Project-specific content 


can also be added to 
— References these sections 


— Roles/Responsibilities 
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ePlan Builder - Home 


Systems Center 
Charleston 


Space and p 
Naval Warfare Pp | B qd 
Systems Center e ad n U e [i ’ 
Charleston Electronic CMM1T@ Compliant Documentation Application 
""New vloagd ‘s Save ‘Quit * Help Sponsored by the Directoy of Engineering Operations (O9K) - Michaal Kutch 


‘olup 


WELCOME to the $8C-Charleston ePlan Builder (EPB) 


EPE ¥2.1 


Keith Jones Version 7.1 
ePlan Buiider 
Customer Liaison 


This application will guide you through the process of creating documentation for your project that is compliant with the 
EnvsiPe usta CMMI" and SSC-C policy and process rannals, 


Liaison 
Tlos release of ePlan Buulder wall now support the Systems Engmeerig Plan (SEP), click here to see the other documents 
that EPB supports. 


Back to CompWeb | —_Docuruents that are in the process of being built will be held for 90 days. Afterward that, they will he punged fiom EPB. 
; Omee a document is generated, it will be held for 24 hours to allow the user to store it m the appropiate CII repository. 
After thas 24-hour penod, EPR wall purge the document. 


) Technical Support 


PROUECT SETLUPF 3) 


Another product of excellence developed by TECHSOFT, Inc. _ 


- | seppanner 
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EPB - Project Setup 


Systeme Center Space and ' 
= aod Plan Builder © 


Charleston Electronie CMMI®™ Compliant Documentation Application 


*’ Help Sponsored by the Director of Engineering Operations (O9K) - Michael Kutch 


¥ Nome ~ Project Setup ~ Document Selup ~ Tulle »  Genercle ¥ 


Project Information 


This section will allow you to enter project information. The information entered here, 
while not included within the generated documents does have a bearing on the text 
and options for a given document. 


Please select the Project Type. 


@ Product 
© Service 


Please select the Product Type. 


Contact 


EPB v2.1 
Keith Jones 
ePlan Builder 

Customer Ligison 


[~ Software 


¥ Hardware 


Email Customer 
Liaison 


Technical Support 


Back to CorpwWeb 


Please select the project effort. 
= = SEIPartner 
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EPB - Project Roles 


Systems Center 
ston SPAWAR Space and } 
a b/ 4 shevel wecrre Mimcl =) Fly siUl| (else 


Charleston 


Electronic CMMI®™ Compliant Documentation Application 


Sove Sponsored by the Director of Engineering Operations (O9K) - Michael Kutc 
a 


v Home ~ Project Setup - Documern! Setup ” Build - Genersic 
Project Roles 


The project has primary personnel that comprise the Management, Development and 
Testing team. Please enter the following roles for this project. 
External Stakeholders: 
Please enter the code and name of the Program Sponsor 


Code;[None | Name:[Wone 


Please enter the code and name of the Program Manager 
Code:[52a_] | Name:PoannaShirey SCS 
Please enter the code and name of the Primary Requirements Provider 


Code:fos sd Name:|Dan Green, FIT Project Office 


Please enter the code and name of the CCB Chairman 


Code:[None ])Name:[None 


Please enter the code and name of the Senior Manager 


Codes2_ | Name:[kenpible 


Email Customer 
Liaison 


Technical Support ; ; 
Required Project Roles 


Please enter the code and name of the Project Leader 


Code:[524. sd Name:|Joanna Shirey 


Back to CorpWeb 
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EPB - Select Tasks for each Role 


s s Center SPAWAR Space and ; 
— bf system Center Mam o) od Ee Lp i elO Tiel otg 
Charleston Electronic CMMI™ Compliant Documentation Application 
Sponsored by the Director of Engineering Operations (O9K) - Michael Kutc 
Generote JS 
oat? -t. eee 12 
Project Leader Tasks 
The Project Leader is responsible for establishing and maintaining the project plan. 

Please identify the specific responsibilities of the Project Leader, 
[¥ Coordinates all activities of the prime contractor and subcontractors 
[¥ Assigns specific responsibilities to subcontractors [PP GP 2.4] 
J¥ Discusses technical issues from the Government with subcontractors 
j¥ Discusses technical issues from the subcontractors with the Government 
j¥ Manages the project cost and schedule [PMC 1.1] 
JV Resolves any inconsistencies in the requirements [PMC 2.2] 
[¥ Mitigates project risks [PMC 1.3] 
j¥ Manage and resolve corrective actions [PMC 2.2] [PMC 2,3] 
JV Provides prime contractor and subcontractor work products and 

deliverables to the Government 

Please enter any additional specific responsibilities of the Project Leader, 

Task 

—————————— 
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SPAWAR Work Breakdown Structure (WBS) 
W, in a Project Management Plan 


Systems Center 
Charleston 


Choose the WBS Source 
[ABC =| 


@ Add Previous Fiscal Year Add Fiscal Year $F 
000 Leadership/Management 2007 


001 Leading 2007 lis00 sk 


O02 Management 


003 Personnel Management Activities 


004 Communications 


100 Project Management 
110 Management Documentation 


111 Programming ® Budgeting 


112 Program Planning Documents 


113 Acquisition Documents 
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VE Risk Identification in PMP 


Systems Center 
Charleston 


Risks 


This page allaws you to enter a list of known or expected risks, The severity of the risks 
and the mitigation approach for each should be identified. Please use the table below to 
identify the major risks associated with the project. 


i) Click for more information about risks 


Risk Categary Impact/Concern Level Mitigation Approach 

[Schedule | Products are required by _# [High 5 Be prepared to provide draft = 
the customer by 10/1/06 + materials if development of = 

Risk Category Irmpact/Concern Level Mitigation Approach 

[Quality s] Will products be ready for _* [ Medium = Provide technical data ta contractor _* 
10/15/06 ina condition * in accordance with schedule with = 

Risk Category Impact/Concern Level Mitigation Approach 


[Technical zl Ability ta get teh = [High 7 Interact directly with the satellite = 
technical ata fram the manufacturer te obtain the technical + 
Acted Vere liems = 


PMP may also reference a more comprehensive Risk Management Plan 
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Measurement & Analysis Plan* 


pio eh nl Gost is a measure within the Financial Performance category that measures the cost for 


activities, events, and products. The measure provides an easy-to understand view of 
the budget. Comparison of planned and actual cost data provides insight into significant 
and repetitive cost changes at the activity level. 


While more detailed cost information provides more insight into the project's total cost, 
until the project personnel have achieved a certain level of proficiency in estimating 
costs, it is recommended that the cost data should be captured at a level commensurate 
with this level of experience. 


Collection and Storage 


Identify the level of detail for capturing cost data 
[Project Level =] 
Please select how the Project Leader will report contract costs from the list below. If the 


Project Leader is not responsible for managing contracts, select "Project". 
[Project =] 

Identify who will provide the actual cost data: 

Project Leader * 


Identify the tool to be used to collect cost data: 
[psaendemacs f==SOCSC™C*~C<=;<C~S*é‘;:*~™S 


Identify how often the actual cost data will be collected: 
Monthly bd 


Analysis Procedures 


Identify how often the cost data will be analyzed: 
Monthly bd 


Identify the cost alert threshold: 
[os% ~! 

: = seiParner 
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Systems Center 
Charleston 


=> seiPartner 
“WECHSOFT 


Technical Software Services, Inc. 


Metrics Reporting 


Measurement & Analysis Plan? 


Please enter the Reports to be generated during the Measurement and Analysis process: 


Milestone Dates 


Report Identifier Measures Periodicit Delivery Method 
Process Performance [Monthly = Ef oo 


Report Identifier Measures Periodicit Delivery Method 
Process Performance [Monthly a Hard Co 


Milestone Dates 
Cast 


Report Identifier Measures Periodicit Delivery Method 
Process Performance [Monthly a Hard Co 


Add Wore lkems <> 
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Systems Engineering Plan (SEP)? 


SEP format follows the DoD SEP Preparation Guide 


Next Life-Cycle Phase 


The SEP requires that the program’s acquisition history and life-cycle phase @ be 


discussed, describing the top-level, technical process used in each life-cycle phase. This 
Next Life-Cycle Phase section should give an overview of the next planned life-cycle 
phase as well as summarize the process activities that are expected to be finished during 
the next life-cycle phase. 


Please enter text discussing the Next Life-Cycle Phase of the program. 


This description should give an overview of the planned SE process and should have 
more detail than the historical life-cycle processes completed. It should include how the 
technical process will be integrated into the life-cycle model and summarize the process 
activities that are expected to be finished during the next life-cycle phase. 


Life-Cycle Phases (in hierarchical order): 


. Concept Refinement 
. Technology Development 

. System Development and Demonstration 
. Production and Deployment 

. Operations and Suppe 


=> seiPartner 
ECHSOFT = 


Approved for release to the public - 12 Oct 2006 


| 


Technical Software Services, Inc. 


NAS Pensacola 
OSP Survey 


PROJECT SETUP 


Design Considerations 


This section describes any design considerations that must be integrated into the 


25 engineering design effort including any special constraints that must be considered, 
—— Please enter any design constraints, 

PEERS BUSTION These design constraints are any special considerations that must be taken into 
EEE account before they are integrated into the project during the engineering process. The 
HISTORY : 
ae text should also describe the basis for these design constraints and how the technical 
CAPABILITIES authority is going to be engaged in considering and integrating these constraints. 
System 

Capabilities Some examples of design constraints are as follows: 

Certification 


Requirements 


Design 
Considerations 


-« The system shall be able to operate using the three phase power available on 


board a ship. 
- The system shall be able to fit into a standard 19" rack, 


Systems Engineering Plan (SEP) 


SE 
SHEA ZEETENE While these constraints look like requirements, they are not system requirements 
INTEGRATION ; 
=== because they do not specify what the system must do, nor do they specify how well 
ENGINEERING the system must perform a capability; they constraint the possible solutions by limiting 
PROCESS the choices available to the engineers, and are therefore design requirements that 
INTEGRATION constrain the solution space. 
The nature of the SEP requires more open input text fields, but 
EPB helps by providing elaborations and examples for the user 
=> SEIPartner 
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= seiParner 
ECHSOFT 


Technical Software Services, Inc. 


Trade Studies 


SEP —- Planned Trade Studies 


This section should include a brief description of the process used to determine trade-offs 
between various attributes of the program (e.g., between requirements and design). 
Information about how trade studies are addressed within the organization will be 
automatically embedded into the document. To view the embedded information about how 
trade studies will be addressed, click the "Click to view the embedded trade studies text" 
link below. 


@ Click to view the embedded trade studies text. 


Trade studies will be addressed in accordance with the SSC-C Technica! Solutions 


Process Manual and SSC-C Decision Analysis and Resolution Process Manual where the 
development of alternate solutions, selection criteria and trade processes are discussed, 


The actual trade studies to be performed on the program will be captured and listed in the 
control below. 


Please enter the trade studies that will be conducted on this program. 
Trade Study 


Research on OSP topologies 


Trade Study 


Research on different conduit installation 
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EPB Output — a Plan 


Systems Center 
Charleston 


|Table of Contenis 


N6S296-93-PMD-00014 | 
Anenst 18,2006 


1.2.1 Plarming forthe Project 
1.22 Monitormg and Controllmg the Project 
3 Products Produced by the Project 
Project Management Plan (PMP) 4° Definitions, Acronyms and Abbreviations 


1.41 Acrormpms and Abbreviations 
For 142 Definition 
MARSOC West SCAMPT CER (593) 
1.5.1 Government References 


1.52 Non-Govemment references 
1.53 Project References 


dugust 18,2006 


22:1 
222 
D295 = Pinpect Leader isss.ccssscccessitsasescstscvesteoee 
2.24 Quality Assurance 
2.25 Configuration Manager 
Prepared Wy! 2.26 
Space and Naval Warfare Systems Center, Chaddeston. aay 


2.28 
(SSC-C) 


P O Bex 190022 
North Charleston, SC 29419-5542 


Approved yt: Matic Reviand (593) (Pate: Auaet93, 3005 


. Project Funding and Sources 
. Risks and Issues 
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EPB Output - CMMI® Compliance Matrix 


Systems Center 


Charleston N65236-593-PMP-0001-v1 


August 18, 2006 
PROJECT PLANNING 


SSC-C 
CMMI®-SE/SW | CMMI®-SE/SW Level 2 PP Process 593 
Goal/Practice Process Area Manual PMP 
Number Project Planning (PP) Paragraph Paragraph 


Establish Estimates. Estimates of project 
planning parameters are established and 
maintained. 


Estimate the Scope ofthe Project. Establish Lad 
and maintain a top-level work breakdown 3 
structure (WBS) to estimate the scope of the Appendix A 
project. 


Establish Estimates of Project Attributes. 
Establish and document estimates of the 
attributes of the work products and tasks. 


Define Project Life Cycle. Define the project 
life cycle phases upon which to scope the 
planning effort. 


Determine estimates of Effort and Cost. he 
Estimate the project effort and cost for the 124 
attributes of the work products and tasks Appendix A 
based on estimation rationale. 


Develop a Project Plan. A project plan is 
established and maintained as the basis for 
managing the project. 
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VE EPB - Technical Specs 


Systems Center 
Charleston 


¢ Architecture 
— Web-based application, with supporting database 


¢ MS SQL Server® 2000 Relational Database Management 
System (RDBMS) 


— Web architecture: Active Server Page, MS .NET Framework® 1.1 
(ASP.NET) 


— Programming Language: MS Visual Basic® .NET (VB.NET) 
— Scripting Language: HTML, Javascript 


— Master Page engine that uses only one Active Server Page (ASP) 
that dynamically retrieves required information (questions, client 
answers, document template text, etc.) 


¢ Development - Incremental life-cycle model 


=> SEIPartner 
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VE. The Second Wave - ML2 to ML3 


Veneteston 
e Shift Focus of Process Improvement Strategy from 
“Implementing CMMI®” to “Executing Sound Systems 
Engineering” 


e Educate Project Managers on What’s Expected 


e Improve Project’s Planning and Documented 
Processes 


¢ Provide additional Training, Workshops and WBTs 


¢ Incorporate ISO, Lean Six Sigma (LSS) and Balanced 
Scorecard (BSC) Initiatives 


= = sEIPartner 
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Ww, Provide Additional Training 


Ss s Center 
Charleston 


¢ All employees need a basic understanding 
of process improvement Over 1600 


: ‘ Tae thVatelerctecs 
— What is their process now? Trained 
— How to add value by formalizing and Total attendance 
| | over 3800 * 


improving their process (based on 
Organizational process) 


°¢ All Pane teams need to fully understand 
the CMMI model (all processes, all levels) 


— To understand all of the best practices 
and maturity levels 


— To comply/prepare for DoD and NAVY 
policy 


* This accounts for some employees attending more than one course 
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SPAWAR p 


a7 4 rovide Workshops 


Systems Center 
Charleston 


¢ All project team members and supporting personnel need to 
know how to perform the standard processes and best 
practices required 


— How to do good SE, CM, PPQA, Planning, Measurement, Risk, 
VER/VAL, etc. 


¢ To properly prepare for and complete an assessment or 
appraisal, key project team members need to map the project 
work products to the practices assessed. 


SEI 
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W, Training Architecture 
F dati f 
cirnionst | | AWB | me 


Core SSC-C project 
and engineering 
processes 
(Level 2 and 3) 


gr cine ; Quality Engineering Requirements Analysis ) 
available on-site classes Configuration Mgmt 


} 


= Existing 
L_] Existing - revise 
L_] New - develop 


WecHsort : MB New - buy 
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Prepare Projects 
for BSC or SCAMPI 


Training and Tools 


Quality Engineering Requirements Analysis ) 
Configuration Mgmt 


eS 
1 —— 
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[==] Existing 
L] Existing - revise 


LJ New - develop 
[J New - buy 


ee SE 101 Web-Based Training 


Charleston 
¢ Introduction to Systems Engineering WBT 


— 10-module web based training 


— Closely aligned to SSC-C SE Process, SE 
Fundamentals Course, and ISO/IEC 15288 


— Includes hotlinks to referenced documentation 
¢ SSC-C Process manuals, policies, standards 
— Extensive branching for more detail 


Technical 
hee\_ Architecture 


ee 


Domain wa 
Product Lines Z 
Sysiem Architecture ? 


System 
Engineering }— 
Process 


os 
7 Originating Yo 
\ Requirements / \ 


ssc-c 7 
d Standard Processes | 
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VE Topical WBTs 


Charleston 
¢ Developing web-based training courses in specific 
topics 
¢ Architecture Development WBT - completed 
— Introduction to Architecture Development and DODAF 


— Designed to educate and promote value of system architecture to 
non-architects and new engineers 


— Tests for understanding 
¢Risk Management WBT 

— Risk identification 

— Analysis tools and techniques ioc a 

— Mitigation planning ew ' >) 

— Risk monitoring LS 
¢Requirements Development _| rei 


One Architecture - Three Views 


Operational 
View 
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VE Introduction to Software Engineering 


Systems Center 
Charleston 


¢ Similar format to the Systems Engineering 
Fundamentals 


— 3 days, primarily lecture 
— Aligned with the SSC-C Software Development Process Manual 


Communication 


Engineering | 


¢ Course Outline 
— Intro to Software Engineering 
— Roles 
— Software Engineering Practices 
— Software Development Process 
— Software Maintenance 
— Managing Software Projects 


Modeling 


Planning 


Practices | 


Deployment 


Coding 
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SPAWAR Engineering Project & Process 
WV, Management Worksho 


Systems Center 
Charleston 


¢ Multi-session workshop oriented “how to” class 
¢e What is a good process? Is my process good? 
¢ How to generate project plans 
— What makes a good PMP, CM Plan, QA Plan... 
— How to use ePlan Builder 
— Hierarchy of plans (Based on level 2 or level 3 goals) 
¢ Configuration Mgmt 
— Are my Configuration Items (Cl’s) and Change Control adequate? 
¢e PPQA 
— How to execute a process review and work product review 
e Measurement and Analysis 
— Are my measures measurable? 
¢ Requirements Management 
— Traceability - simple to complex 
¢ Monitoring and Control using Reviews 
Wecusort 
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VE. The Second Wave — ML2 to ML3 


Veneteston 
e Shift Focus of Process Improvement Strategy from 
“Implementing CMMI®” to “Executing Sound Systems 
Engineering” 


e Educate Project Managers on What’s Expected 


e Improve Project’s Planning and Documented 
Processes 


e Provide additional Training, Workshops and WBTs 


¢ Incorporate ISO, Lean Six Sigma (LSS) and Balanced 
Scorecard (BSC) Initiatives 


= sEIPartner 
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VE. A World Class Engineering Organization 


Systems Center 
Charleston 


SW 
Process 
Standard 


SE 
Process 
Standard 


Qms 
ISO 9001/2000 


Provides a 
Benchmark for Best 
Practices 


Implements 
a QMS 


Lean Six Sigma 


Improves 
Stable 
Processes 
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VE SO and LSS Initiatives 


Systems Center 
Charleston 


¢ ISO 9001/2000 is an internationally recognized standard for Quality 
Management Systems (QMS) Principles. 


— ISO requires the Organization to focus upon organizational processes, the 
Customer, and continual improvement. 


— ISO requires documented processes and procedures that complement and 
supplement existing processes and procedures. 


— Itis acommon sense, documented system focusing on consistency, reliability, 
and improving the way businesses operate. 


— ISO provides elaboration of the CMMI QA best practices, giving you the detail to 
implement the Quality Management System 


¢ Lean Six Sigma is an Internationally recognized process improvement 
methodology that combines both Lean and Six Sigma. 


— Lean minimizes waste and improves process flow. 

— Six Sigma improves process capability and eliminates process variation. 
— Lean Six Sigma is an aid in improving stable processes and process flow. 
— Lean Six Sigma is applied to stable, mature processes. 
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VE. ISO - CMMI Relationships 


*Gharleston 
Hi-level view ISO: Management 
Responsibility 
ISO: CMMI: ISO: Measurement, 
Quality LY OPF, OPD, RD, Analysis & 
Management PMC, OPP, QPM; Improvement 
System GP'2.1, 2.2, 2.0, 2.4, CMMIe: 
ISO: Resource 2.6,.2:7,210;3.1 PMC, PPQA, 
CMMIe: Management MA, CM, REQM, 
OPF, OPD, CMMIE2: RD, SAM, OPF, 
PP, PPQA, PP, OT, ISO: Product VER, VAL, OID, 
CM, SAM; OEI: Realization OPP, QPM, CAR; 
GP 2.1, 2.2, GP 2.3, 2.5 CMMIEe: GP 2.1, 2.2, 2.4, 
2290697 REOM, RD, TS, Pl, 2.6, 2.8, 2.9, 3.2 
2.8, 2.9, 3.1, QY MA, QPM, VER, VAL, 
3.2 OPD, PP, PMC, IPM, 


CM, SAM; 

GP 2.1, 2.2, 2.3, 2.4, 
2.6, 2.7, 2.8, 2.9, 2.10, 
3 
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M7 2 CMMI®, LSS and ISO Comparison 


Systems Center 
Charleston 


CMMI LSS ISO improvement framework comparison 


ithe — Kc — Kile — Ail 1 — Al 


(ileal ks Ks CART 


@ © OQ O 
(= | " (es 
K 1 | an ad 
2 S| 

©0000 © 

ea 

q = 

| A \ 

L/ \\ 


Systems Center 
Charleston 


Stakeholder 


Customer 


Fully Netted Force 


Provide Composeable 
C4ISR Capability 


Balanced Scorecard Strategy 


S1 = ‘Fully Netted in Three’ 


2 Deliver Real-Time 
Business Data 


Provide Organized 
C3 Accessibility 


Customer Focused 


Li 
Balance the Right Readiness 
With Right Cost 


Mission Focused 


IC1 
Manage Customer 
Relationships 


Develop a World — 
Class Systems Engineering 
Capability 


M1 
Perform Core 
Work In-House 
IM2 
Enable Capacity 
for Context Work ~~ 


Wecusort 


Technical Software Services, Inc. 


Learning 


| HCL, ign Competencies wit 
Strategies 


Achieve 

Strategy-Focused, 
Shared Awareness, & 
Aligned Effect 


Enable an Empowered, 
Collaborative & 
Synchronized Environment 
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SPAWAR 
v, 


Systems Center 


Charleston 


prspectives Objective Headline 


IC1: Manage 
Customer 
Relationships 


Customer 
(Internal) 


IM1: Perform Core 
Work In-House 


Mission 


IM2: Enable Capacity 
for Context Work 


IP1: Developa 
World-Class 
Engineering 
Capability 


Process 


IP2: Developa 
World-Class 
Business Operation 


=> sEIPas 
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Objective Statement 


Construct and align our enterprise processes to insure 
maximum customer involvement. 


Apply our limited government resources to the mission- 


critical and strategy-valued functions 


Maintain a sufficient level of contract capacit' 
support our outsourcing and out-tasking req 


Establish standard engineering processes 
that provide a qualitative competitive edge in 


the global market 


Establish standard business processes that 
complement our engineering processes and provide 


a competitive edge in the global market 


Measure 


On-line Customer Profile - Percent 
of customers for which we have 
current tasking and profiles that are 
enterprise wide accessible 


Government Man-Hours on Core 
Work - Percent of government man- 
hours spent on core 


contract ceiling with actual burn rate 


A. Technical Maturity 
Assessments. Average 


_tatings from assessments, _ 


B. Level Two/Three/Four 
Projects - Count the number 
of department projects self- 
assessed at Level 2/3 


A. Core business Processes 
Captured and Documented - Ratio 
of the business processes within 
the functional areas in Enterpri 
Proce MI gemen chitecture 


that are defined and documented 
with proper interfaces identified 
and controlled / facilitated 


B._ Core Business Process 
Improved - Dollars saved or costs 
avoided by the improvement of 
business operations processes, 
with their organizational interfaces 
accommodated by the Enterprise 
Process Management Architecture 


Balanced Scorecard Based on CMMI® 


Purpose 


To drive the customer focus thru all 
engineering and business processes 


To increase percentage of core 
government labor hours to total 
government man-hours 


Context Ceiling Capacity - Availab | Monitor available ceiling to ensure it 


exceeds expected work 


Drive the level of 
implementation of common 
Systems Engineering practices 


‘Oo promote standardization of 
processes and greater efficiency of 
overall business operations 


To promote standardization of 
processes and greater efficiency of 
overall business operations 


VE Incorporate ISO, LSS, BSC and CMMI® 


Systems Center 
Charleston 
¢ Implementation NUTT) em cieareye xe dar= 
— sr all to complement each Process Knowledge Base 
other 


— Promote consistent 
implementation from 
Command down 


« Benefits 


— Objective evaluation and QA 
available to all projects, large 
or small 


— Cost sharing for QA or 
process reviews on an as- 
used basis 


— Central ieee of issues and 
lessons learned for sharing 
across division and 
department 
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W, The Second Wave — ML2 to ML3 


Charleston 
¢ Develop internal “self-assessment” process for measuring 
ongoing implementation of Maturity Level 2 processes 
¢ Populate EPO/CMMI® Website with ‘best examples’ 
¢ Implement Tailoring Guidelines 
¢ Implement SSC-C Measurement Repository 
¢ Implement ML2-to-ML3 Action Plans 


¢ Continue to Measure and Communicate Progress 


e Maintain Momentum and Commitment to Goals 


WecHsort 
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VE Internal “Self-Assessment” Process 


Systems Center 
Charleston 


¢ Good starting point for 
project to understand 
what’s required 


e Teaches difference 
between Direct and 
Indirect evidence and 
the need for both 


¢ Serves as the BSC 
Measure of project’s 
progress towards 
ML2/ML3/ML4 


e Artifacts are reviewed 
for compliance 


= => seiPanner 
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lal! 


Type a question forhelp +» . @ X 


‘8 +/BI 0 EBlap 
a 


Ey Microsoft Excel - Worksheet in Bull CM sourcesafe temp sst278,doc 


‘) Fle Edt View Insert Format Tools Data Window Help 
GMB AFAR) 9-8 EA) @ Bh testentonan 
Ay he 

Project Planning (PP 

roject Monitoring and Control (PMC 
Confiquration Management (CM 

Process and Product Quality Assurance (PPQA) 
Requirements Management (REQM)} 

Supplier Agreement Management (SAM) 


Measurement and Analysis (MA 


Specific Practice 
Process Area 


Project Monitoring and Control 


Monitor Project Against Plan, Actual performance and progress of the 
[project is monitored against the project plan, 


SP 1.1 | Monitor Project Planning Paranneters. Monitor the actual values of the project 
planning parameters against the project plan, 


SP1.2 | Monitor Comumutments. Monitor comunitiments against those dentified in the 
project plan. 


|: ES if 


NUM 


Ready 
stat} (Jz Merse..o| 8) reser. | (vss scu.| ste (Ra.| a] str Rea. |i] caso. «3A 3490 
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ae 4 : Populate Website with ‘Best Practices’ 


Systems Center 
Charleston 


¢ Improved/formalized process to 
populate EPO/CMMI® Website 
with ‘best examples’ 


— EPO submittal form to submit via 
email 


— Direct and timely response that 
artifact was received 


— Process for SME to evaluate 
artifact and recommend posting, 
holding for future date, not posting 


— Process for project to submit 
artifact with Do Not Post notice 


— Process to notify when submittal 
IS posted or reason for not posting 


SEIF 
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VG. Improved Content and Usability of Website 


Systems Center 
Charleston 


e Gave Website a “Facelift” 


¢ Populated it with many “Good 
Samples” of ML2 and ML3 
Plans and SOPs for projects to 
use 


¢ Helps Institutionalize GP 3.2 for 
all Process Areas 


— Collect work products, measures, 
measurement results, and improvement 
information derived from planning and 
performing the process to support the future 
use and improvement of the organization’s 
processes and process assets. 


oT 


Technical Software Services, Inc. 


Wecuso 
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W, The Second Wave — ML2 to ML3 


‘Vhereeon 
e Develop internal “self-assessment” process for measuring 
ongoing implementation of Maturity Level 2 processes 


¢ Populate EPO/CMMI® Website with ‘best examples’ 
¢ Implement Tailoring Guidelines 

¢ Implement SSC-C Measurement Repository 

¢ Implement ML2-to-ML3 Action Plans 

¢ Continue to Measure and Communicate Progress 


e Maintain Momentum and Commitment to Goals 


= sEiPariner 
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W, Tailoring Guidelines 


Systems Center 


Charleston EXCerpts from the “SSC-C Developing Project Processes SOP” ..... 


In order to initiate the Tailoring Process, a Project/Task Leader follows the guidance 
Hille in the SSC-C Tailoring Guidelines and the SSC-C Integrated Project 
anagement Process Manual to establish and manage a project according to an 
integrated and defined process that is tailored from the organization’s set of 
Sere processes. The SSC-C Tailoring Guidelines can be found on the EPO 
website. 


There are six activities within the Tailoring Process, described in detail in the SSC-C 
Tailoring Guidelines: 


Establish the project profile 

Select project processes from the SSC-C Set of Standard Processes 
Complete the Tailoring Form 

Complete Waiver Request, if applicable 

Complete Peer Reviews of Tailoring Forms and Waiver Request, if applicable 


Department Head reviews Waiver Request 


The results of the completed Tailoring activities are used in 
conjunction with the SSC-C Integrated Project Management 
Process Manual to establish the Project’s Defined Process. 
The basis for establishing the SSC-C Project’s defined 
process is SSC-C’s standard processes. The SSC-C 
Systems Engineering Process Manual, SSC-C Software 
Development Process Manual, and the SSC-C Software 
Maintenance Process Manual provide the detailed activities 
and tasks needed to successfully implement the process 
requirements provided in the SSC-C Level 2/3 Process 
Manuals. 
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Measurement Repository 


Ss s Center 
Charleston 


¢ Built SSC-C Measurement 
Repository (OMR) for 
projects to use for 
managing their projects 
— Capturing standardized cost, 


schedule, and process ject 262 
performance wrens tes 99920 3: ws 


¢ Implementation included | BAB nea te 
hands on training in : 
reporting and using OMR 


e Laying groundwork for 
higher maturity 
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VE ML2 to ML3 Action Plans 


Systems Center 
Charleston 


e PI Plan to address ML2 
findings and ML3 actions 


e Lessons learned from 
SCAMPIs 


¢ Incorporate Tailoring 
Guidelines 


e Provide metrics to OMR 


¢What’s ML3 all about? 
— Peer Reviews 
— SEMPs 
— DAR Plan 
— RSKM Plan 
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W, The Second Wave — ML2 to ML3 


Charleston 
e Develop internal “self-assessment” process for measuring 
ongoing implementation of Maturity Level 2 processes 
¢ Populate EPO/CMMI® Website with ‘best examples’ 
¢ Implement Tailoring Guidelines 
e Implement SSC-C Measurement Repository 
¢ Implement ML2-to-ML3 Action Plans 


¢ Continue to Measure and Communicate Progress 


e Maintain Momentum and Commitment to Goals 


7 => seiPanner 
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VG. Measure and Communicate Progress 


Systems Center 
Charleston 


¢ Continuing similar approach to Maturity Level 3 

¢ Successful ML3 Program (with 4 projects) in July 2006 
e Another project achieved CL3 in 16 Process Areas 

¢ 4 more projects with planned SCAMPIs in 2006 


¢ Continuing to use Continuous Representation and 
Equivalent Staging for early successes 


¢ S2E Newsletter is a great means of communicating 
status and success 
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VE. Maintain Momentum/Commitment to Goals 


Systems Center 
Charleston 


©2007 
— Conduct interim Maturity Level 3 appraisals on projects 
— Correct findings and strengthen institutionalization 
— Conduct Command Maturity Level 3 appraisal in April 2007 
— Incorporate new version of CMMI model (V1.2) 
— Incorporate IPPD (Integrated Product and Process Development) 
— Mentor projects to Maturity Level 3 


¢ 2008 - 2009 
— Begin Maturity Level 4/5 implementation 
— Establish/Refine/Compose organizational and project measures 


— Increase collection of project, process, and organizational 
measurement data 


— Conduct interim Maturity Level 4 appraisals on projects 
‘WecHsort : 
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VG. Summary? 


Systems Center 
Charleston 


¢ Decided on Approach — use CMMI® for Process 
Improvement and Measuring Progress 


e Using extensive research, determined the ‘Critical 
Success Factors for Implementing CMMI®” 


¢ Built Plan of Action 
e Advertised Early Successes 
¢ Implemented Plan Successfully for Phase 1 


¢ First SPAWAR Systems Center to achieve CMMI® 
Maturity Level 2 in April 2005 
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VE. Summary? 


Systems Center 
Charleston 


¢ Following Plan for Phase 2 — CMMI® Maturity Level 3 
¢e Focus on System Engineering Excellence 


¢ Continue to apply “Lessons Learned” 


— Findings from SCAMPIs 


e Peer Reviews, Effectiveness of DAR, Senior Management 
Involvement, PPQA 


— Feedback from using the SSC-C Organizational Assets 


— Feedback for using the SSC-C Organizational Measurement 
Repository 


— Feedback from ePlan Builder (MR Process in place) 


=> SEIPartner 
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VE Summary? 


Yetariesion 
e Leveraging from the first program in SPAWAR 
Systems Center to achieve CMMI® Maturity Level 3 
— Interviews and pictures in S2E Newsletter 
— Sample ML3 Documents on Website 
— Findings from SCAMPI 


¢ Continuing to prepare for the Command Maturity 
Level 3 appraisal in April 2007 


e Laying groundwork for higher maturity 
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Systems Center 
Charleston 


Summary* 


«e Successes 


— April 2005 Command Achieved CMMI® 
aturity Level 2 as certified by Software 
Engineering Institute 
— July 2006 first program to achieve 
CMMI® Maturity Level 3 


— 1S‘ SPAWAR Systems Center to 
achieve these levels 


¢ Aggressive SE Program 
¢ Industry Standards 

— Systems Engineering 

— Software Engineering 
¢ Best Practices 


fTEAN_| . 
cmmi 3X — 
ety Aik — World-Class SE Program 
ee ____ == | — Support Command Balanced Scorecard 
e Automated Tools EPO Website -— = — April 2007, Command to achieve 
— ePlanBuilder ON Ses: CMMI® Level 3 
— eWBS conniehe spawannaunmilennall 


¢ Training — 1,600+ 
— SE Fundamentals - 305 Ay 
— Web-Based Training courses ss +. 
« SSC-C PI; Intro to SE; Arch. Dev. =~ 


SPAWAR 
Vv, 


Systems Center 


— Thank you ! 


Any Questions ? 


Contact Information: 


Michael T. Kutch, Jr. Sandee Guidry 

SPAWAR Systems Center Charleston Technical Software Services, Inc. 
Email: michael.kutch@navy.mil Email: sdguidry@techsoft.com 
Phone: 843-218-5706 Phone: 850-469-0086 
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Achieving the Vision 


CMMI Technology Conference 
Luncheon Keynote Address 


November 14, 2006 


Mark D. Schaeffer 
Director, Systems and Software Engineering 
Office of the Under Secretary of Defense (AT&L) 
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The initial vision for CMMI was to integrate the 
competing maturity models and provide a 
framework for more consistent process 
improvement 
¢ Cause integration of the functional disciplines within 
Organizations and across programs 


e Increase systems engineering and software process 
maturity as organizations migrate from the sun-setting 
CMMs to CMMI 


* Extract: 2004, 2005 CMMI Conference Keynotes 2 


a  )) Have We Lost Sight of the Goal?* 


e The end goal of CMMI Is to provide a model for 
continuous process improvement to achieve: 


— Reduced cycle times 
— Meet cost and schedule targets 
— Improved quality 


—Common Systems Engineering and Software model 


* Extract: 2004, 2005 CMMI Conference Keynotes 3 


( yy How We Got Where We Are* 


¢ CMMI Sponsors opted to pursue staged and continuous 
models to preserve legacy 
- SW-CM\M, staged - SECM, continuous 


e Acquiring organizations do not have full understanding of 
how CMMI is intended to be used 
- What a specific level at the enterprise level actually means to an 
acquisition program 
- That the process and people evaluated to obtain a level are not 
necessarily applied to their program 


- Achievement of a specific level may or may not have meaning to any 
given acquisition program 


* Extract: 2004, 2005 CMMI Conference Keynotes 4 


(73) Efforts to Define the Problem 


¢e NDIA report to DoD - CMMI issues, Feb 2005 
¢ NDIA Workshop on CMMI Use in DoD Programs, 
Sept 2005 
— Explored issues, and identified a set of 
recommendations 
¢ Spring ‘O6 Government Review and Assessment 
— Validated Workshop findings 


— Refined and augmented issues, developed 
recommended actions 


— Directly influenced CMMI v1.2 
¢ Ongoing: DCMA data call 


Summary of CMMI Problems 
DoD Sponsor Report 2006 


e Programs execute at lower maturity levels than their organizations have 
achieved and advertised 


¢ High-maturity practices are not consistently applied at the project level 
after contract award 


¢ How to ensure new projects will incorporate CMMI processes 


Appraisal sampling procedures — how to ensure adequate coverage of 
the organizational unit 


Appraiser quality — training, consistency 

e Lack of agreement on what constitutes Levels 4 and 5 
¢« Need to converge to a single representation 

Content of appraisal disclosure statements is lacking 
Inadequate training and education for acquirers 

e Should CMMI be used for source selection 


| ___ Whatis the resolution of these issues? —__| 6 


Pa CMMI Government Assessment: Spring 2006: 
MLA 8 Major Issues for Resolution 


1. High Maturity 
e Lack of consistency and agreement on what 
constitutes levels 4 and 5 
2. Integrity 


¢ If certified, how can programs be performing ata 
lower level? 


3. Organizational Commitment 


¢ Commitment and ability to implement processes on 
new projects 


4. Acquirer Education 
e Misunderstanding and misuse of CMMI by Acquirers 


£ ~~ % CMMI Government Assessment: Spring 2006: 
WLP 8 Major Issues for Resolution 


5. Two representations 

e¢ Staged and Continuous representations 
6. Model Complexity and Size 

e Balancing content (700 pages) with ease of use 
7. New Constellation Strategy 

¢ Impacts of the new architecture on original goal 
8. Intended usage of CMMI 


¢ Current application of CMMI against original and 
future goals 


Improvements Implemented 
in CMMI v1.2 


e “Level for Life” - ELIMINATED 


¢ Organizational Commitment 


— Added new goal and 2 practices to CMMI v1.2 to address 
commitment to processes at project start-up 


¢ Appraisal Sampling 
— Developed new sampling rules 
— Precise definition of sample size and organizational coverage 
e Appraisal Disclosure Statement (ADS) 
— Added specific sampling information to enhance transparency 
— DoD Contractor ADSs will be posted for Government acquirer access 


Improvements Implemented 
in CMMI v1.2 


¢ High maturity appraisers 
— Established certification requirements for appraisers 
— Oral exams being given to all high maturity appraisers 
— High maturity training course under development — Oct 07 release 
¢ Guidebook for Acquirers 
— Provide concise information to acquirers on supplier use of CMMI 
— Expected release January 2007 
¢ Appraisal Integrity 
— Lead Appraiser cannot be from the appraised business unit 


— Lead Appraiser certification of sample, and appropriateness of 
Level 4/5 practices 


— SEI is conducting appraisal audits 
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Remaining Opportunities 
..and some Questions 


¢ Revisit Levels 4 and 5 
— Do we need something else to define high maturity? 
e Lean the Model and the Appraisal Method 
— Eliminate cumbersome material included for legacy reasons 
— Eliminate staged representation? 
¢ Evaluate Constellation strategy 
— Will Constellations result in stovepipes? Do they make sense? 
e Assess next generation process improvement 
— Should CMMI be used for source selection? 


— Given worldwide adoption, is the CMMI Governance Structure 
appropriate? 


| __ Need your ideas and participation = 11 
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e Evaluate changes to the CMMI v1.2 product suite 
to ensure improvement goals are really being 
met 

¢ Monitor constellation evolution 

— Pilot CMMI-ACQ model 
— Assess implementation 

¢ Continue to capture the right knowledge in the 

CMMI Guidebook for Acquirers 


e Leverage DCMA for actual performance 
monitoring 


a) | Reinforce the basics 


¢ Continue focus on refining what CMMI was intended to 
achieve 


— Does the current product suite allow us to achieve those objectives? 
e Make sure that v1.2 changes bring value added to the user 
— Assess the value of each change 


e Ensure changes facilitate achievement of the CMMI 
objectives 
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Questions? 


What We Learned Along the Way 


Where We Were: 
Slides from My 2004 Luncheon Address 


What We Have Accomplished, 
And What’s Left To Do 


NDIA Workshop and Summit on 
CMMI Use in Acquisition 


¢ Held on September 7 & 8, 2005 


e Issues Discussed 

— Background on how organizations approach CMMI appraisals and 
why 

— Use of Appraisal Disclosure Statement by acquiring organizations 

— Formal guide to CMMI Usage for DoD 

— Training for DoD Acquisition Organizations in the use of CMMI for 
DoD 

— Government lack of understanding of need for mature SE content 
and practice 

— Specifying or requiring CMMI in RFPs 
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( ih | CMMI Revitalization 


¢ DoD has invested heavily in the CMMI project as an 
important Systems and Software Engineering (SSE) initiative 


e Problems have surfaced with CMMI 


¢ This briefing will describe these problems, actions being 
taken, and future plans: 

— Where We Were 
e Slides from My 2004 Luncheon Address 

— What We Learned along the Way 
e Workshops, Assessments, Data Calls 

— What We Accomplished and What’s Left to Do 
e Resolution of Some Issues for v. 1.2 
e Moving Forward 
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(ce 4) ) DCMA Survey and Data Collection* 


Survey conducted in response to OUSD (AT&L) request: 


e “How well do suppliers maintain their CMMI/CMM assessed levels during 
contract execution?” 


Data collected on 30 Programs (first in a quarterly collection process) 
ACAT Levels reported 

e ACAT — IAC — 1 program 

e ACAT IC — 8 programs 

e ACAT ID — 18 programs 

e ACAT Il — 3 programs 
Claimed maturity levels (MLs) 

e 7 CMM ML3 

e 1CMM ML4 

e 10 CMMI ML3 

e 2CMMI ML4 


* rs 
© 10 CMMI ML5 Excerpt from DCMA initial 


findings briefing, 3 Nov 06 
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“(eaat) I Initial DCMA data reveals... 


e Programs are adhering to organizational processes 
— 7/% of programs find the supplier performs their defined 
processes with minor non-compliance 
— The majority of programs do tailor 0-10% of their 
processes 
e Reasons for process non-adherance 
— Primary reason is schedule 
— Cost, resources, and customer impacts are also causes 
e There does not appear to be a link between higher 
maturity levels and program performance 
— No correlation between maturity levels and CPI/SPI 


Issue Area 1: 
High Maturity/Level 4&5 


e Issues: 


— Lack of agreement on what constitutes Levels 4 and 5 


— High-maturity practices are not consistently applied at the project level after 
contract award 


e Resolution: 
— Certification of high-maturity appraisers is now in place 


e July 06 workshop on high maturity training determined the requirements 
for which the lead appraisers have to provide evidence in order to do 
high maturity appraisals 


e Previous courses, oral questions, etc. 
— Body of Knowledge on high maturity will be developed and available by 
October 2007 
— Lead Appraisers must certify that level 4 and 5 appraised subprocesses map 
to organization’s business objectives 
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L-- Issue Area 2: 
a( 7) = Integrity - Programs Execute at Lower 
=“ _—s©@ Levels than Achieved and Advertised 


e Findings that lead to Integrity Issue: 
— Issues with appraiser quality — training, consistency, independence 
— Content of Appraisal Disclosure Statements (ADS) lacking 
— Appraisal sampling inconsistent, and influenced by appraised 
organization 
e Resolutions: 
— CMMI v1.2 training upgrade: face-to-face training with focus on integrity 
— Eliminated “level for life’— now 3 year limit 


— DCMA developing a survey related to CMMI appraisal ratings and 
program performance to begin to measure the problem 


— CMMI v1.2 updates to the ADS (separate chart) 
— CMMI v1.2 updates to Sampling (Separate chart) 


23 


Issue Area 2: 
Integrity - continued 


e Issue 


— Appraisal sampling inconsistent, and influenced by appraised 
organization 


— Appraisal sampling representation of the organizational unit 
e Resolutions: 
— Precise definition of the sample 


e Organizational scope: name, type, location 


¢ Organizational Unit Coverage: size, application domain, geographical 
breadth, project type expressed in percentages of total organizational unit 


e Projects excluded and rationale 
— Lead Appraiser certification that focus and non-focus projects are 
representative of organization 


— Lead Appraisers must come from an organization other than the 
business unit being appraised 
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Issue Area 2: 
Integrity - continued 


e Issue: 


— Content of Appraisal Disclosure Statement (ADS) is not representative of 
CMMI appraisal data 


e Resolution: v1.2 ADS requires improved level of detail 
— Provides details on appraisal sample 
e Organizational unit, projects, domains 
— Provides Lead Appraiser certifications 
e Project sampling is representative 
e Level 4/5 certifications are based upon practices that represent organizational 
business goals 
— All appraisals performed after 28 Nov 2006 must use ADS v1.2 (includes 
CMMI v1.1 appraisals) 
— DoD contractor ADS’s will be posted (website thd) for Government acquirer 
review 
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Issue Area 3: 
Organizational Commitment 


e Issue: 
— How to ensure new projects will incorporate CMMI processes 


e Solution: 


— Added new goal and two practices to Operational Process Focus 
(OPF) PA to stress deployment of processes to projects. 


— Added text in Integrated Project Management (IPM) PA to emphasize 
having a defined process at project start-up 


— CMMI Guidebook for Acquirers will discuss need to address project 
level implementation with developers 
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Issue Area 4: 
CMMI Guidance for Acquirers 


e Issues 
— Inadequate training and education for acquirers resulting in 
misrepresentation and misuse of CMMI 
e Solution 


— CMMI Guidebook for Acquirers 
e Due out after the first of the year 


e Will address how CMMI should be used by acquirers, how to interpret 
appraisal results, how to treat CMMI throughout the lifecycle 


— Development of the CMMI-Acquisition Constellation 
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CMMI TECHNOLOGY CONFERENCE & USER GROUP 
AN ENTERPRISE APPROACH TO PROCESS IMPROVEMENT 


VADM WALLY 
MASSENBURG 


COMMANDER, 
NAVAL AIR SYSTEMS 
COMMAND 


GLE FLEET DRIVEN MEASURE OF SUCCESS 


AIRCRAFT AND CARRIERS READY FOR TASKING AT REDUCED COST 


14 NOVEMBER 2006 


S OUTLINE 


¢ OUR ENTERPRISE JOURNEY 


¢e AIRSPEED SUCCESS STORIES 


¢ NAVY ENTERPRISE: THE WAY AHEAD 


OUR ENTERPRISE JOURNEY 


S} THE DILEMMA 


20-YEAR-OLD AIRCRAFT ARE COSTLY TO MAINTAIN ... 
MAINTENANCE MAN-HOURS PER FLIGHT HOUR 
INCREASING ... 
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... AND WE CAN’T AFFORD TO BUY 
THE NUMBER OF NEW AIRCRAFT WE NEED 


10 


AVERAGE AGE 


FYO7 | FYO8 | FYO9 | FY10 | FY11 


FY99 | FYOO | FY01 | FY02 | FY03 | FY04 | FY05 | FY06 


71 125 144 150 164 
re 119 143 
105 140 
128 


THE PROBLEM: 


WE NEED TO BUY AN AVERAGE OF 190 NEW AIRCRAFT EACH YEAR 
TO MAINTAIN OUR AVERAGE AIRCRAFT AGE 


98 00 02 04 06 O08 10 12 
FISCAL YEAR 


S NAVAL AVIATION ENTERPRISE 
LEAN SIX SIGMA (LSS) JOURNEY 


¢ THE YEARIS FYO1... 
— CURRENT READINESS ...$1B DEFICIT 
— FUTURE READINESS ...NEED MORE AIRCRAFT 
¢ CNO EXECUTIVE BOARD DIRECTION... DECEMBER 2000 
— SINGLE PROCESS OWNER FOR NAVAL AVIATION 
— FIX CURRENT READINESS 
— AFFORD FUTURE READINESS 


CEO 


WE NEEDED ro NStusion 

© ACH; CHANGE 

Ev Ou 

(= DIFFEREN, a a BEHA Vior 
COMES 


S$ NAVAL AVIATION ENTERPRISE 


NAVAL AVIATION VISION 
RIGHT FORCE... 
RIGHT READINESS ... 
RIGHT COST... 
RIGHT TIME... 


CFFC | CNAF 


- CNAVRES 
- CNATRA 
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« 
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ASN(RDA)(DASN(AIR), PEOs, 
NAVAIR, NAVSUP(ICP, DLA), 
NMPC, CNET(NPDC) 


N78 | N43 | HQMC 


... TODAY, AND IN THE FUTURE 


CONFIG. MGR: ERIC BADERTSCHER, NAPO, (301) 757-7691 


S ENTERPRISE PRINCIPLES 


e APPLY A PROCESS PERSPECTIVE 


¢e UTILIZE A SET OF CONSISTENT, INTEGRATED, 
AND HIERARCHICAL METRICS 


e ENSURE FULL AND CONSISTENT TRANSPARENCY 
OF DATA AND INFORMATION THROUGHOUT 


¢ ESTABLISH AND MAINTAIN PROCESS DISCIPLINE 
THROUGHOUT 


¢ ESTABLISH AND MAINTAIN ACCOUNTABILITY FOR 
ACTIONS AND RESULTS THROUGHOUT 


¢ APPLY AN INTEGRATED GOVERNANCE 
STRUCTURE 


A DELIBERATE, DISCIPLINED PROCESS TO ACHIEVE AVIATION UNITS READY 


FOR TASKING AT THE RIGHT COST... TODAY AND IN THE FUTURE 


BRIEF DATE: 14 NOVEMBER 2006 
CONFIG. MGR: ERIC BADERTSCHER, NAPO, (301) 757-7691 


OPERATING AS AN ENTERPRISE 


SINGLE PROCESS OWNER 
SINGLE FLEET-DRIVEN METRIC: 
AIRCRAFT AND CARRIERS READY FOR TASKING 
AT REDUCED COST (CONTINUING TO MATURE) 
VALUES - WHAT WE BELIEVE 

— FLEET READINESS 

“COST-WISE” (LESS $s) 

— TIME ON WING (LESS STUFF) 

— SPEED (LESS TIME IN MAINTENANCE) 

— PEOPLE (CONTINUOUS IMPROVEMENT) 

THREE MAIN FOCUS AREAS: 

— READINESS: TODAY, TOMORROW, AND FUTURE 

— TOTAL FORCE READINESS 

— COST MANAGEMENT 


BRIEF DATE: 14 NOVEMBER 2006 
CONFIG. MGR: ERIC BADERTSCHER, NAPO, (301) 757-7691 


S$ APPROACH 


NEED TO DEFINE: 


¢ POLICIES / PROCESSES 
NEEDED TO OBTAIN OUTCOME; LASTLY...... 


¢ BEHAVIORAL CONSTRUCTS / CONOPS 
TO ACHIEVE COST-WISE & RESPONSIVE OUTCOMES. ..... 


A BEHAVIORAL MODEL FOR THE GREATER GOOD 


DON LEAN SIX SIGMA (LSS) JOURNEY 


CULTURAL 
DOD DON SENIOR INSTITUTIONALIZATION 
LEADERSHIP DIRECTS 
USE OF LSS 


ASN RD&A DIRECTS USE 
OF LSS 


NAVY & USMC INDUSTRIAL BASE 
SHOWS OUTPUT FROM LSS dt 


so) Mw Ke) A=) -0) 


TRANSACTIONAL 
PROJECTS YIELD 
OUTPUT 


NAVSEA 
TASK FORCE 
LEAN 


NAVAL AVIATION apps 
ENTERPRISE (NAE 
1990 ee) COMMENCE 
NAVAIR H 
DEPOTS COMMENCE een renee 


ISO 9000 200 
2000 
NAE AIRSPEED 
1999 ny 
LEAN EVENTS ON THREE 
os AIRCRAFT CARRIERS 


_STRIVING FOR: “LSS - BUSINESS AS USUAL” 


NAVAL AVIATION ENTERPRISE AIRSPEED 
LEAN, SIX SIGMA, TOC 


eee 
FOUR PROGRAMS: ONE GOAL, ONE METRIC 


1999 | 2003 +=-| 2004 | 


DEPOT ENTERPRISE NAVAIR 

* DEPOT PRODUCTION | - FLEET-WIDE REPAIR - CORPORATE | 

PROCESSES SITES AND COMPETENCY 

PROCESSES PROCESSES AND 

- LED BY DEPOT 

AIR-6.0 oe LED BY 0-6 ESC OTHER PRODUCTIVITY 

ROLLOUT TO 3 SITES O-I-D + SUPPLY descent 

CHAIN - LED BY CCBU/ EDB 
GUIDANCE 
- LINKS TO NAVRIIP 


¢ ENABLED BY 
¢ ROLLOUT TO 59 CORE TEAM 
SITES 


¢ LINKS WITH 
ENTERPRISE & 
DEPOT AIRSPEED 


SINGLE, FLEET-DRIVEN METRIC 


“AIRCRAFT AND CARRIERS READY FOR TASKING AT 
REDUCED COST... 


TODAY AND IN THE FUTURE” 


2005 
NAVICP 


* CORPORATE | 


COMPETENCY 
PROCESSES AND 
OTHER PRODUCTIVITY 
INITIATIVES 


¢ LED BY NAVSUP 


GUIDANCE 


¢ ENABLED BY 


DEPLOYMENT TEAM 


¢ LINKS WITH 


ENTERPRISE & 
DEPOT AIRSPEED 


PROCESS IMPROVEMENT 


¢ EQUIPMENT / 
FACILITIES 


° PEOPLE CLASSIC 

¢ SUPPLIERS “ADD RESOURCES” 
ie ii eentetg aa APPROACH 

« TECHNOLOGY 


¢ FINANCING 


rim ¢e PROCESSES 
| + METHODS CYCLE-TIME APPROACH - 
MY: cuvture PROCESS, 
~. BEHAVIOR PROCESS, 


¢ STRATEGY PROCESS 


BRIEF DATE: 14 NOVEMBER 2006 
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S NAVAIR AIRSPEED 
INVESTMENT AND COST DATA SUMMARY - FY06 


LABOR / TRAINING / TOOLS DATA FORECAST FOR FY06 - TYPE 1/11 SAVINGS DATA THRU 01 JUL 06 


INVESTMENT 
¢ GEORGE GROUP CONTRACT (NAVAIR ELEMENTS ONLY): $ 12.30M 
e AIRSPEED TOOLS (INCLUDES TRAINING FACILITIES / 

POWER STEERING / MINITAB / IGRAFX SOFTWARE): $ 0.89M 
e BLACK BELT / GREEN BELT - LABOR / TRAVEL: $ 15.80M 


TOTAL $28.99M 
NAVAIR AIRSPEED VALIDATED SAVINGS 
© TYPE! $19.1M 
© TYPE II $15.3M 
TOTAL $34.4M 


AS OF 01 JUL 06: RETURN ON INVESTMENT (ROI) = 1.2 


AIRSPEED INVESTMENT AND RETURN ($M) 


° FY07-09 INVESTMENT IN PEOPLE / TOOLS / TRAINING IS “EXAMPLE BASELINED” ~$30M — (FY05-06 ACTUAL 
BUDGET) 


- TYPE! SAVINGS ARE CUMULATIVE OVER THE FYDP ... IF $19.1M IS SAVED IN AS A RESULT OF A PROJECT 
COMPLETED IN FYO6, THEN IT IS ALSO SAVED FY07 THRU FY09.. .. SAVINGS ESTIMATES FOR FY07-09 ARE 
BASELINED AT $19.1M TO SHOW CUMULATIVE POWER OF AIRSPEED . . . FY09 [FY06-$19.1M + FY07-$19.1M(EST) 
+ FY08-$19.1M(EST) + FY09-$19.1M(EST) = $76.4M] 


¢ TYPE Il SAVINGS ARE NOT CUMULATIVE OVER THE FYDP... VARY YEAR-TO-YEAR BASED ON PROJECT 
SELECTION 


¢ EXPECTATION IS THAT BASELINED SAVINGS INCREASE ... AIRSPEED IS SELF SUFFICIENT IN OUT YEARS 


$80 ; 


TYPE I SAVINGS 


@ AIRSPEED INVESTMENT 


@ TOTAL TYPE | SAVINGS FY06 

$60 - @ TOTAL TYPE | SAVINGS FYO7 (FYO6 + 
07EST) 

W TOTAL TYPE | SAVINGS FY08 (FY06 + 
O7EST + O8EST) 

@ TOTAL TYPE | SAVINGS FY09 (FY06 + 
O7EST + O8EST + 09EST) 

@ TYPE II SAVINGS FY06 


= $40 | AIRSPEED INVESTMENT 


rrr ) Pe fe fen fe pe w TYPE Il SAVINGS FY07 


O TYPE Il SAVINGS FY08 


“INVESTMENT” 


@ TYPE Il SAVINGS FY09 


IS CONSTANT 


FY06 FYO7 FY08 FYO9 


TYPE II SAVINGS 


BRIEF DATE: 14 NOVEMBER 2006 
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S ACHIEVE ENTERPRISE 
COST EFFICIENCIES 


AIRSPEED PRINCIPLES OF OPERATION: 


e PLACE THE RIGHT INVENTORY AT THE RIGHT 
SPOT 
¢ TRADE SPEED FOR WIP 
¢ OPTIMIZE MATERIAL / LABOR CONSUMPTION 
¢ DRIVE VARIANCE OUT OF THE ENTERPRISE 
— THEORY OF CONSTRAINTS 
— LEAN MANUFACTURING 
— SIX SIGMA 


SHIFT REFLEX BEHAVIOR FROM “BUY MORE STUFF AND PEOPLE” 
TO “BUY MORE SPEED” 


BRIEF DATE: 14 NOVEMBER 2006 
CONFIG . MGR: ERIC BADERTSCHER, NAPO, (301) 757-7691 | 5 
FILE NAME: 00_CMMI_Denver_14Nov06.PPT 


WHAT WE’VE LEARNED 


..» OR WHAT I WISH ’'D KNOWN FROM THE START” 


A TOTAL ENTERPRISE APPROACH IS REQUIRED FOR BEST 
RESULTS 


CHANGE REQUIRES COMMITMENT / WILLINGNESS TO 
DRIVE 


THROWING MONEY AT BROKEN PROCESSES IS PRETTY 
CLOSE TO A CRIME 


ESTABLISHING THE RIGHT METRICS (OUTPUT / CUSTOMER) 
IS ESSENTIAL 


NOT EVERYTHING CAN BE FIXED INTERNALLY—OUTSIDERS 
CAN IDENTIFY, DRIVE AND ACCELERATE CULTURAL 
CHANGE—NEED HELP IN MANAGING PROCESS AND 
DISCIPLINE IN EXECUTION 


PROCESS DIMENSION: AS MEASURED BY: 


e PROCESS SPEED: INVENTORY ON HAND 
e PROCESS QUALITY: FIRST-PASS YIELD 
e PROCESS EFFECTIVENESS: TURNAROUND TIME 


e PROCESS EFFICIENCY: COST PER UNIT 


AIRSPEED SUCCESS STORIES 


700 


600 


500 


400 + 


DAYS 


300 


200 


100 


CYCLE-TIME REDUCTION 


EA6B 


PRE/POST LEAN 


TAT 


RW02 & RW03 
600+ DAYS 
605 


SDLM AFC89-89A CWS 
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PM AFC89-89A CWS 


GPRE-TAT @ POST TAT 


SDLM AFC82-894 CWS 


45,000 


40,000 


35,000 


30,000 + 


25,000 


20,000 


15,000 


10,000 


5,000 


VITH 


) S 


FUNDED vs. 


ACTUAL LABOR HOURS 


EA6B 
PRE/POST LEAN 
MANHOURS 


42,726 


34,637 


36,953 


30,682 


SDLM AFC89-89A CWS PMI AFC89-89A CWS 


SDLM AFC82-89A CWS 


@ PRE-FUNDED MHRS 


H PRE-LEAN ACT MHRS 0 POST-FUNDED MHRS 


O POST-LEAN ACT MHRS 
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AIRSPEED IMPROVEMENTS TO P-3 ORION AIRCRAFT 
MAINTENANCE & REPAIR, NAVAIR DEPOT JAX 


SBT LEAD: RICK THEILACKER AIRSPEED CHAMPION: BILL UPDEGRAFF 
DEPOT AIRSPEED INITIATIVE HOSHIN GOALS 
* BECOME “DEPOT OF CHOICE” BY REDUCING WIP AND CYCLE SAFETY * HIGHEST SAFETY OF ALL GOVERNMENT 
TIME, IMPROVING ON-TIME DELIVERY AND IMPROVING QUALITY INDUSTRIAL FACILITIES 


* ZERO AVIATION MISHAPS 


5S ENTIRE FACILITY AND CREATE SINGLE-PIECE FLOW MOVING- QUALITY ¢ HIGHEST QUALITY OF ANY MRO FACILITY 
LINES UTILIZING CELLULAR WORK CENTER DESIGN 


FACILITIES IMPROVEMENTS - FACILITIES PAINTING, LIGHTING, CUSTOMER » ON-TIME DELIVERY TO OUR CUSTOMER 
ELECTRICAL SERVICE UPGRADE, ROOF REPAIR, ARTISAN BREAK 


AREA, OFFICE SPACE FINANCIAL + TOTAL COST REDUCTION 


100% OF WORKFORCE TRAINED IN AIRSPEED 
PROCESS ~~ * CONTINUOUS PROCESS IMPROVEMENT THRU 
AIRSPEED 
¢ OPERATIONAL MATURITY LEVEL 3.0 
AIRSPEED ACCOMPLISHMENTS PEOPLE * EMPLOYER OF CHOICE 
HaGP SCeEEE ¢ TRAINING, EMPOWERMENT AND INVOLVEMENT 


DEFINING WORK CELLS BY TRADE 
SETTING UP QTS SPOT FOR FUELED A/C 


AIRSPEED ACTIVITIES 


¢ DEFINING SHORED AND NON-SHORED SIDES OF HANGAR FOR Apr-04 Lean Deployment Commenced 
EASE OF A/C MOVES May-04 Fuel Leaks - Quick Turn Spot, Hazmat Carts 
NEXT... Jun-04 Foam Install - Quick Turn Spot, Hardware Cage 
° E BTeOuiGEr Acco RUCASSESSMENTS Oct-04 G3 Metal Tank Repairs - Metals Cell established 
¢ KITTING Dec-04 | How Infrastructure - Induction Scheduling, REINTE! 
¢ KAN BAN FOR MINI-SHOPS Process, Kitting 


Jan-05 Disassembly - Disassembly Cell established, Kitting 
continued, Metals/NDI Process 


Area of FY04 Pre- FY06 Post- Mar-05 Assembly - Assembly Cell established, G2 Sheet Metal, 
Improvement AlIRSpeed AIRSpeed Minishop 
On Time Delivery Apr-05 Assembly! Finals - Kitting Design 
ee © Apr-06 AlRSpeed Champion assigned 


BRIEF DATE: 14 NOVEMBER 2006 


F404 ENGINE I-LEVEL 
CONSOLIDATION 


EFFORT BEGAN EARLY ‘04 


IDENTIFIED EXCESS CAPACITY 
(57%) 


SIGNIFICANT SAVING PROJECTED 
9-5 SITES (FY05) 
5-3 SITES (FY06) 


$161M SAVINGS THROUGH FY11 


SPOTLIGHTED THE POTENTIAL OF 
PROCESS IMPROVEMENT 
ACTIVITIES 


" AIRSPEED, LEAN, TOC, 
SIX SIGMA 


NAE-WIDE APPROACH 
NOV 04: NAE PROCESS LAUNCHED 


ENGINE REPAIR CONSOLIDATION AND 
PRODUCT ENTERPRISE TEAMS (PET) 


THE PET PROCESS BRINGS: 


= STANDARD PROCESSES & METRICS 


= PRIORITIZATION OF WORK AND 
INVESTMENT 


TO MAXIMIZE 


7 RELIABILITY, CYCLE TIME, 
INVENTORY, COST, AND SAFETY 


= INTEGRATION OF OTHER TOOLS 
7 (AIRSPEED, BLACK BELTS, ETC.) 


= FOUNDATION FOR NAE-LEVEL 
PRIORITIZATION 


A NEW WAY OF DOING BUSINESS 


i 


PARDON OUR DUST WHILE WE GROW! 


SBT LEAD: JOHN CRUMPLY AIRSPEED CHAMPION: CHRIS KOPP 


FUTURE F-18 INNER WING SHOP, NAVAIR DEPOT JAX 
(GREENFIELD PROJECT) 


CHALLENGE OPPORTUNITY 
¢ HANGAR 122 DEMOLITION 83K SQ FT ¢ MFG BB EVENT RECOVERED 13K SQ FT 
= CURRENT WING SHOP 29K SQ FT * LEAN EVENTS RECOVERED 10K+ SQ FT 
* FLIGHT LINE 34K SQ FT * AIC KITTING EVENTS 
" PAINT BOOTH 20 SQ FT ¢e AIC WIP REDUCTION 
* FUTURE FIA-18 WING WORKLOAD? * REMOVED AIRCRAFT ASKARS (WIP 
REDUCTION) 


RESULTS: RECOVERED OVER 23K SQ FT 


GOODNESS: FOOTPRINT REDUCTION RESULTING FROM PREVIOUS AIRSPEED EVENTS 


IS ALLOWING US TO ABSORB LOSS OF HANGAR 122 WITHOUT NEW CONSTRUCTION 


G5 CURRENT READINESS CFT SUCCESS 


FA04 |-Level 
Consolidation 


" Effort began early ‘04 
" Identified &&cess capacity (57%) 
Significant saving projected 
9-5 Sites (FY O05) 
4-3 Sites (FY OB) 
Pib M savings through Fy'11 
" Spotlighted the potentia of 
process improvement activities 


AlRSpeed LEAN, TOC, 
Sk SIGHTA, 


" NAE wide approach 
implemented F al ‘04 


= EFFICIENCIES: FY05-EXECUTED FHP TO 6+2 DESPITE INITIAL $122M SHORTFALL 
= GENERATED ADDITIONAL $160M BY UNDERSPENDING TO PLAN TO COVER 
ADDITIONAL GNFPP HOURS, 2™° FUEL INCREASE, DEPOT NNOR, PACFLT SHIP 


OPERATIONS 


CULTURE CHANGE: CROSS-FUNCTIONAL DELIVERY OF COST-WISE READINESS 


Millions 


Cross Functional Effects Example: 
BCM-4 Expenditures 


e FUTURE READINESS: BOUGHT 111 AIRCRAFT VICE 100 IN FY05... CHALLENGE 


IS TO BUY RIGHT NUMBER OF AIRCRAFT... 


BRIEF DATE: 14 NOVEMBER 2006 
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NAVY ENTERPRISE: 
THE WAY AHEAD 


NAVY ENTERPRISE 


INDIVIDUAL 
WARFIGHTER ENTERPRISES DEMAND 


SIGNALS 
as 
\V/ C\ 


{_] 
= 
PRODUCTIVITY DRIVERS: 


— PRIORITIES — COMMON METRICS 


PROVIDERS 


& ENABLERS REQUIREMENTS 


MPT&E 


AT&L 


INSTALLS 


HEALTH 
CARE 


S&T 


OUTPUT = 


READINESS 
COST 


VENO | Wes 
ASN BDA) IXONW (Ss) 


PROVIDERS RESOURCES 


— BEHAVIORAL CHANGES — INTEGRATED CAPABILITIES 
— SINGLE PROCESSES / OWNERS — TRANSPARENCY OF INFORMATION 


BRIEF DATE: 14 NOVEMBER 2006 
CONFIG. MGR: ERIC BADERTSCHER, NAPO, (301) 757-7691 2A 


FILE NAME: 00_CMMI_Denver_14Nov06.PPT 


S$ DESIRED NAVY ENTERPRISE OUTPUT 


Vv READINESS OVER COST TODAY 


VY READINESS OVER COST TOMORROW 


Vv READINESS OVER COST IN THE FUTURE 


ACHIEVED THROUGH BEHAVIORAL MODEL (INTERDEPENDENT CONCEPT OF 
OPERATIONS): 


— NAVY ENTERPRISE (GOVERNANCE BOARD): 


SENIOR NAVY STRATEGIC DECISION FORUM FOCUSED ON IMPROVING PRODUCTIVITY FOR CURRENT 
AND FUTURE READINESS THROUGH INTEGRATION OF SUPPORTED WARFIGHTER ENTERPRISES 


— WARFIGHTER ENTERPRISES (FIVE SUPPORTED TEAMS; LED BY “SUPER TYCOMs”): 
COLLABORATIVE TEAMS FOCUSED ON DELIVERING WARFIGHTING CAPABILITY TO NAVY COMPONENTS 
AND COMBATANT COMMANDERS; AND INCREASING PRODUCTIVITY ACROSS THEIR DOMAIN AT REDUCED 
COST 


PROVIDERS / ENABLERS (SUPPORTING ELEMENTS; WITH DESIGNATED LEADS): 
OPERATE AS PROVIDERS / ENABLERS TO MANAGE VALUE STREAMS (PEOPLE, DOLLARS, AND STUFF), 
SUPPORTING TYCOM-LED WARFIGHTER ENTERPRISES, WITH LINKED AND COMMON PROCESSES / 
METRICS 


DOMAIN: DOLLARS, PEOPLE, & STUFF ASSOCIATED WITH EACH WARFIGHTER ENTERPRISE 


DEMAND SIGNAL: DERIVED FROM THE WARFIGHTER ENTERPRISES (I.E., READINESS REQUIRED 
AND NO MORE) 


ENTITLEMENTS: WHAT’S NEEDED, WHEN, HOW MUCH, AND NO MORE 
OUTPUT: READINESS OVER COST 


BRIEF DATE: 14 NOVEMBER 2006 
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CRITICAL NAVY ENTERPRISE 
PROCESSES AND BEHAVIORS 


e IDENTIFY DOMAINS AND ASSIGN SINGLE PROCESS OWNERS 
¢ ASSEMBLE THE RIGHT ENTERPRISE TEAMS AND GAIN COMMITMENT 


¢ OPERATE IN SUPPORT OF A SINGLE FLEET-DRIVEN METRIC (WHAT THE 
ENTERPRISE VALUES) 


— AGREEMENT ON SCOPE, OUTPUTS, AND LINKED METRICS 

— TRANSPARENCY OF DATA TO PROMOTE TRUST AND MONITOR PERFORMANCE 

— SHARED KNOWLEDGE ON ISSUES AND KEY PROBLEMS AFFECTING THE DOMAIN 
— RECOGNIZE, NURTURE AND RESPECT TECHNICAL AUTHORITY 

— IDENTIFIED ENTITLEMENTS (WHAT’S NEEDED, WHEN, HOW MUCH, AND NO MORE) 


e AGREE ON DESIRED OUTPUT (E.G., READINESS OVER COST), WITH FOCUS / 
TRADE-SPACE INVOLVING CURRENT AND FUTURE READINESS 


¢ OPERATE WITH DISCIPLINE, GOVERNANCE, AND A REGULAR (TIMELY) DRUMBEAT 


e BASELINE EVERY DOLLAR, ALL THE PEOPLE, ALL THE STUFF, AND ALL THE 
CAPABILITY WITHIN THE DOMAIN, WITH ASSIGNED ACCOUNTABILITY FOR 
OUTCOMES 


e ESTABLISH ENTITLEMENTS; CONTINUALLY MEASURE GAPS-TO-ENTITLEMENT 
e REMOVE BARRIERS TO PRODUCTIVITY 


NAVY ENTERPRISE CONSTRUCT 
COCOM sevice 


MEETINGS 


STRATEGIC GUIDANCE 
AND PROGRAMMING 
NAVY DECISIONS 


ENTERPRISE 


N8/ FMB 


OPS / READINESS PLANNING 
AND REQUIREMENTS 
PRIORITIZATION 


READINESS 
NTERPRISES 


bred OPNAV N8 


INTEGRATED ENTERPRISE 
DROCESSES AND READINESS 
EXECUTION 


PLATFORM 
ENTERPRISES 


PROVIDERS Behe ites 


SPONSORS 
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HOW SENIOR LEADERSHIP CAN HELP: 
AREAS OF EMPHASIS 


¢ COMMIT TO THE CHANGE —- MAKE IT LAST THROUGH 
LEADERSHIP TURNOVER 


— INCLUDE IN PERFORMANCE EVALUATIONS / FITNESS REPORTS (FITREPs) 
— INCENTIVIZE GROUP PERFORMANCE GOALS FOR SENIOR LEADERSHIP 


— INCLUDE PRODUCTIVITY IMPROVEMENT TRAINING IN LEADERSHIP 
DEVELOPMENT PROGRAMS 


¢ PARTICIPATE IN THE EFFORT 
— PROVIDE EXECUTIVE SPONSORSHIP FOR PROJECTS 
— GET TRAINED AS A GREEN BELT 
— DEMAND DATA AND METRICS... STAMP OUT OBFUSCATION 


¢ PROVIDE RESOURCES - VISION WITHOUT MONEY EQUALS 
HALLUCINATION 


— PROVIDE STABLE FUNDING TO ENSURE SUCCESS 
— DEMAND VALIDATED RETURN ON INVESTMENT 


LEADERSHIP COMMITMENT AND ACCOUNTABILITY 
MAKE CULTURAL CHANGE A REALITY 


Sa 


=. 


THE FLEET 


QUES 


NAVAL AVIATION ENTERPRISE 
LEADERSHIP 


ine , a = - 
me NEAsNE AVIATION/VISION 
= >. 2 


> ~~ = 


BACKUP 


NAE “SYSTEM” SCM 


STRATEGIC COST MANAGEMENT ENVIRONMENT 


Sa 


AIRCRAFT 
CV(N) soil ADVERSARIES 
WEPS RANGES 
S| M S IMA Beet 
PARTS 


FLYING HOUR 
PROGRAM 


4Aa42memanecengoan 
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NAVAIR SOFTWARE PRODUCTIVITY 
IMPROVEMENT EXAMPLES 


(COMPARISONS OF TEAM’S SW-CMM LEVEL 1 PERFORMANCE VERSUS 
SW-CMM LEVEL 4 PERFORMANCE) 


ie Cost 
Productivity Quality Increase 
(MH/ISLOC) ETE) HoT @9) (Defects/KSLOC @OTRR) 


P-3 (Design Only) 46% 35% 78% 


37% 56% 46% 


34% 68% 45% 


NOTES: 


ABOVE DATA IS PROVIDED TO SHOW GAINS REALIZED BY NAVAIR ORGANIZATIONS THAT HAVE IMPROVED THEIR 
PROCESSES BY MEANS OF CARNEGIE-MELLON SOFTWARE ENGINEERING INSTITUTE CAPABILITY MATURITY MODELS. 


DATA IS BASED ON ACTUAL OFP COSTS UP TO OTRR FOR F/A-18 AND AV-8. P-3 DATA IS BASED ON ACTUAL DESIGN 
COSTS. 


DATA SHOULD NOT BE USED TO COMPARE ONE ORGANIZATION AGAINST ANOTHER, SINCE VARIATIONS IN SOFTWARE 
COMPLEXITIES AND REQUIREMENTS WOULD MAKE SUCH COMPARISONS INVALID; FOR EXAMPLE: 


e REAL-TIME EMBEDDED APPLICATIONS VS. DESKTOP MISSION SUPPORT APPLICATION; 


e LEGACY SOFTWARE DEVELOPMENT (ASSEMBLY LANGUAGE) VS. NEW LANGUAGE DEVELOPMENT 
(C++ OBJECT-ORIENTED). 
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BUSINESS-BASED CMMI APPROACH 
NOTIONAL PROJECT 2 
3 


‘il 


Org = Proj. Proj. Supplier —_Integ. Risk Quant. 
Innov.& Planning Mon.& Agree, Project Mgmt. Project 
DI. Control Mgmt. Mgmt. gmt. 


Capability Level 
o 


Program Management 
¢ ADOPTING CONTINUOUS REPRESENTATION OF CMMI MODEL 
e FOUR STEP PROCESS: 


— 1)DETERMINE WHICH CMMI PROCESS AREAS ARE GERMANE TO THE PROJECT TEAM 


— 2) DETERMINE PROCESS CAPABILITY LEVELS NEEDED TO DELIVER REQUIRED FLEET 
PRODUCTS (AND NO MORE) 


— 3) ASSESS PROJECT TEAM’S CURRENT CAPABILITY LEVEL INEACH PROCESS 
4) PERFORM BUSINESS CASE ANALYSIS TO DETERMINE BEST INVESTMENT AREAS 


¢ WILL DETERMINE CMMI MATURITY VIA EQUIVALENT STAGING AT MAT LEVEL 
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